You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/10/15 07:00:58 UTC
[GitHub] [spark] HeartSaVioR opened a new pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
HeartSaVioR opened a new pull request #30051:
URL: https://github.com/apache/spark/pull/30051
NOTE: Do not merge. If the test passes and we are OK with the direction, I'll cherry-pick these commits one by one manually.
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711493891
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711476999
**[Test build #129975 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129975/testReport)** for PR 30051 at commit [`3a0af10`](https://github.com/apache/spark/commit/3a0af1011a1bf18d6445fd6426b047cffd865d72).
* This patch **fails build dependency tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711475300
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #30051:
URL: https://github.com/apache/spark/pull/30051
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-708945230
This is a proposal on handling backport of #30037 to branch-3.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-709139531
**[Test build #129813 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129813/testReport)** for PR 30051 at commit [`3a0af10`](https://github.com/apache/spark/commit/3a0af1011a1bf18d6445fd6426b047cffd865d72).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-708985797
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34418/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711493878
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34583/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-709144652
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711633947
Following commits are just landed to branch-3.0.
https://github.com/apache/spark/commit/15ed3121994d4c7e6f1db7e0b5b2cea5565b495a
https://github.com/apache/spark/commit/02f80cf293739f4d2881316897dbdcea74daa0bc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711496057
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34585/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-712505206
There're two points here if I understand correctly, so I'll comment each point.
1. Using revert of revert
I never used `revert` to `claim` or `blame` something. That is simply moving the commit out of, nothing else. Same applies to this case, revert of revert. I didn't intend anything, but I'm sorry if you feel I use revert to disagree with your revert. I agree I should cherry-pick the origin commit again instead of doing revert of revert. My bad.
2. Necessity of creating a backport PR
I'm sorry but I disagree, because of the different view on root cause of the issue.
The original commit was wrong because SPARK-32557 didn't land to branch-3.0, despite it was a follow-up of SPARK-32529 which was landed to branch-3.0. While the type of SPARK-32557 is marked as "improvement" (I guess that was the reason it didn't land to branch-3.0), I can simply turn it to "bug" as it actually fixes a bug in real world.
That's a two sides of the coin - unless the improvement doesn't change the behavior, it is somewhat likely possible to fix a bug. The strict rule about when to backport (backport only for a bug type) is really not pragmatic and that should be only enforced when we don't believe others (while that can be still used as a guideline). Is it the case?
In my view I did two works 1) SPARK-32557 is missing in branch-3.0 so I "corrected" it. 2) After that I cherry picked SPARK-33146 to branch-3.0 again as "usual practice" on merging phase. To not break again I made WIP PR to confirm these works don't break anything. I'm not sure I didn't respect some policy here.
Btw, that was my bad I didn't do some verification after cherry-picking and simply pushing. Though that was my bad, I think it's arguable (and probably meaningful) topic that how many verifications can be done during cherry-picking.
IMHO, enforcing to build "without test" on cherry-picked branch is somewhat pragmatic, as it requires around 15+ mins which may be acceptable. That still breaks the flow, but well, we may account it for "responsibility".
Building with test is completely different story - we'll let our development environment be stuck for 3~4 hours, which doesn't seem to be something we want to enforce to mergers. If that is enforced I expect the negative impact on avoiding to port back while it's ideal to port back. (Again this wouldn't happen if we ported back SPARK-32557.)
Always requesting to contributors to make a backport PRs and require them to hang around for more than 4 hours isn't also the solution - I think it's too hard for volunteers (both mergers and contributors). Accounting all of questions I raised, I guess it costs less we tolerate the possibility of test failures and fix it later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-712410420
Hey, @HeartSaVioR . I don't think the revert of revert is a correct way here. At least, if you really want to `revert of revert`, you should make a PR and pass the UT in the community instead of silent reverting of reverting.
02f80cf293 Revert "Revert "[SPARK-33146][CORE] Check for non-fatal errors when loading new applications in SHS""
The way I see this is the following.
1. The original commit was wrong because it's committed without testing.
2. Hence, the revert is legitimate to recover `branch-3.0`.
3. After you tweaks `branch-3.0` yesterday, you cannot claim of `revert of revert`. Instead, you had better land SPARK-33146 as a normal backporting PR with `[3.0]` tag.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711615864
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711476376
**[Test build #129975 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129975/testReport)** for PR 30051 at commit [`3a0af10`](https://github.com/apache/spark/commit/3a0af1011a1bf18d6445fd6426b047cffd865d72).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-709671902
@dongjoon-hyun The tests passed - are you OK with the approach I proposed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711487029
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34583/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711477014
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129975/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711480913
**[Test build #129977 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129977/testReport)** for PR 30051 at commit [`3a0af10`](https://github.com/apache/spark/commit/3a0af1011a1bf18d6445fd6426b047cffd865d72).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711476376
**[Test build #129975 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129975/testReport)** for PR 30051 at commit [`3a0af10`](https://github.com/apache/spark/commit/3a0af1011a1bf18d6445fd6426b047cffd865d72).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-709144652
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711493897
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34583/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-708997921
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34418/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711480913
**[Test build #129977 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129977/testReport)** for PR 30051 at commit [`3a0af10`](https://github.com/apache/spark/commit/3a0af1011a1bf18d6445fd6426b047cffd865d72).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711479566
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-708997909
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711491292
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34585/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-708997909
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-708997872
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34418/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711496065
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-708945466
**[Test build #129813 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129813/testReport)** for PR 30051 at commit [`3a0af10`](https://github.com/apache/spark/commit/3a0af1011a1bf18d6445fd6426b047cffd865d72).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711615864
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711477010
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711613910
**[Test build #129977 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129977/testReport)** for PR 30051 at commit [`3a0af10`](https://github.com/apache/spark/commit/3a0af1011a1bf18d6445fd6426b047cffd865d72).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711496065
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-708946513
Thank you for investigating, @HeartSaVioR ! Let's see.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-712519418
Okay, it seems that we agree to disagree. Thank you for sharing your opinion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711493891
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-712505206
There're two points here if I understand correctly, so I'll comment each point.
1. Using revert of revert
I never used `revert` to `claim` or `blame` something. That is simply moving the commit out of, nothing else. Same applies to this case, revert of revert. I didn't intend anything, but I'm sorry if you feel I use revert to disagree with your revert. I agree I should cherry-pick the origin commit again instead of doing revert of revert. My bad.
2. Necessity of creating a backport PR
I'm sorry but I disagree, because of the different view on root cause of the issue.
The original commit was wrong because SPARK-32557 didn't land to branch-3.0, despite it was a follow-up of SPARK-32529 which was landed to branch-3.0. While the type of SPARK-32557 is marked as "improvement" (I guess that was the reason it didn't land to branch-3.0), I can simply turn it to "bug" as it actually fixes a bug in real world.
That's a two sides of the coin - unless the improvement doesn't change the functionality, it is somewhat likely possible to fix a bug. The strict rule about when to backport (backport only for a bug type) is really not pragmatic and that should be only enforced when we don't believe others (while that can be still used as a guideline). Is it the case?
In my view I did two works 1) SPARK-32557 is missing in branch-3.0 so I "corrected" it. 2) After that I cherry picked SPARK-33146 to branch-3.0 again as "usual practice" on merging phase. To not break again I made WIP PR to confirm these works don't break anything. I'm not sure I didn't respect some policy here.
Btw, that was my bad I didn't do some verification after cherry-picking and simply pushing. Though that was my bad, I think it's arguable (and probably meaningful) topic that how many verifications can be done during cherry-picking.
IMHO, enforcing to build "without test" on cherry-picked branch is somewhat pragmatic, as it requires around 15+ mins which may be acceptable. That still breaks the flow, but well, we may account it for "responsibility".
Building with test is completely different story - we'll let our development environment be stuck for 3~4 hours, which doesn't seem to be something we want to enforce to mergers. If that is enforced I expect the negative impact on avoiding to port back while it's ideal to port back. (Again this wouldn't happen if we ported back SPARK-32557.)
Always requesting to contributors to make a backport PRs and require them to hang around for more than 4 hours isn't also the solution - I think it's too hard for volunteers (both mergers and contributors). Accounting all of questions I raised, I guess it costs less we tolerate the possibility of test failures and fix it later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711632168
I'll go with this approach as this is a simplest way to deal with. Cherry-picking two commits to branch-3.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-708945466
**[Test build #129813 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129813/testReport)** for PR 30051 at commit [`3a0af10`](https://github.com/apache/spark/commit/3a0af1011a1bf18d6445fd6426b047cffd865d72).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711477010
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-712410420
Hey, @HeartSaVioR . I don't think the revert of revert is a correct way here. At least, if you really want to `revert of revert`, you should make a PR and pass the UT in the community instead of silent reverting of reverting.
02f80cf293 Revert "Revert "[SPARK-33146][CORE] Check for non-fatal errors when loading new applications in SHS""
The way I see this is the following.
1. The original commit was wrong because it's committed without testing.
2. Hence, the revert is legitimate to recover `branch-3.0`.
3. After you tweaks `branch-3.0` yesterday, you cannot claim of `revert of revert`. Instead, you had better land SPARK-33146 as a normal backporting PR with `[3.0]` tag.
In general, `Revert of revert` is used when the first revert decision was wrong. Here, it looks like you use it as a claim that your first commit was right.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30051: [DO-NOT-MERGE][SPARK-XXXXX][CORE][3.0] Port back SPARK-32557 and SPARK-33146 to branch-3.0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30051:
URL: https://github.com/apache/spark/pull/30051#issuecomment-711496072
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34585/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org