You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/12 08:39:50 UTC
[GitHub] [spark] otterc opened a new pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
otterc opened a new pull request #30349:
URL: https://github.com/apache/spark/pull/30349
### What changes were proposed in this pull request?
This is a follow-up fix for the failing tests in `YarnShuffleServiceSuite.java`. This java class was introduced in https://github.com/apache/spark/pull/30062. The tests from here should be moved to
`resource-managers/yarn/src/test/scala/org/apache/spark/network/yarn/YarnShuffleServiceSuite.scala`
### Why are the changes needed?
The newly added tests are failing when run with hadoop profile 2.7
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Ran the unit tests with the default profile as well as hadoop 2.7 profile.
`build/mvn test -Dtest=none -DwildcardSuites=org.apache.spark.network.yarn.YarnShuffleServiceSuite -Phadoop-2.7 -Pyarn`
```
Run starting. Expected test count is: 11
YarnShuffleServiceSuite:
- executor state kept across NM restart
- removed applications should not be in registered executor file
- shuffle service should be robust to corrupt registered executor file
- get correct recovery path
- moving recovery file from NM local dir to recovery path
- service throws error if cannot start
- recovery db should not be created if NM recovery is not enabled
- SPARK-31646: metrics should be registered into Node Manager's metrics system
- create default merged shuffle file manager instance
- create remote block push resolver instance
- invalid class name of merge manager will use noop instance
Run completed in 2 seconds, 572 milliseconds.
Total number of tests run: 11
Suites: completed 2, aborted 0
Tests: succeeded 11, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726492836
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726009974
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-725997857
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35591/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-725931417
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] asfgit closed pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #30349:
URL: https://github.com/apache/spark/pull/30349
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-727101593
Thanks for merging it @tgravescs.
Thanks for fixing this @otterc, and thanks for reporting it @dongjoon-hyun !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-727063440
merged to master
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-727066052
Thank you, @tgravescs and all!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726352375
**[Test build #131019 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131019/testReport)** for PR 30349 at commit [`c11f583`](https://github.com/apache/spark/commit/c11f583174d0309637f4daee9c25e2d2a76278e1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726351291
Let's see if this recovers `Hadoop 2.7` on Maven build.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726813558
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-725942091
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-725943142
**[Test build #130985 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130985/testReport)** for PR 30349 at commit [`c11f583`](https://github.com/apache/spark/commit/c11f583174d0309637f4daee9c25e2d2a76278e1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-725954372
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-727038201
**[Test build #131071 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131071/testReport)** for PR 30349 at commit [`c11f583`](https://github.com/apache/spark/commit/c11f583174d0309637f4daee9c25e2d2a76278e1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` test(\"invalid class name of merge manager will use noop instance\") `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726815632
**[Test build #131071 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131071/testReport)** for PR 30349 at commit [`c11f583`](https://github.com/apache/spark/commit/c11f583174d0309637f4daee9c25e2d2a76278e1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-725930735
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-725954193
**[Test build #130985 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130985/testReport)** for PR 30349 at commit [`c11f583`](https://github.com/apache/spark/commit/c11f583174d0309637f4daee9c25e2d2a76278e1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` test(\"invalid class name of merge manager will use noop instance\") `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726795476
yeah I didn't realize that Suite was in different module, it makes sense it put it all in one place.
it looks like ./common/network-yarn doesn't have dependencies for yarn (only Hadoop client) whereas ./resource-managers/yarn does. the tests aren't doing much with yarn it just calling your static function for newMergedShuffleFileManagerInstance and it builds with just the client so it seems like that should be enough. If tests pass I think its fine to merge as it makes sense just to combine into single file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-727062536
I'm not sure what is going on with GitHub Action tests, but based on the Jenkins ones passing, I'm going to merge this. If we find it makes things worse we can revert it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726492414
**[Test build #131019 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131019/testReport)** for PR 30349 at commit [`c11f583`](https://github.com/apache/spark/commit/c11f583174d0309637f4daee9c25e2d2a76278e1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` test(\"invalid class name of merge manager will use noop instance\") `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726852944
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726852944
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726852907
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35675/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-725930735
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726351291
Let see if this recovers `Hadoop 2.7` on Maven build.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-725954372
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-725931417
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726350706
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726352375
**[Test build #131019 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131019/testReport)** for PR 30349 at commit [`c11f583`](https://github.com/apache/spark/commit/c11f583174d0309637f4daee9c25e2d2a76278e1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726106190
it would be nice to add description as to why they need to be moved. or why moving them fixes whatever the failure is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726009974
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726492836
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726146622
Big +1 for @tgravescs advice.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726009988
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35591/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-727038664
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726384961
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35625/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726840457
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35675/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726384975
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726815632
**[Test build #131071 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131071/testReport)** for PR 30349 at commit [`c11f583`](https://github.com/apache/spark/commit/c11f583174d0309637f4daee9c25e2d2a76278e1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
otterc commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-725930315
@mridulm Please take a look
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726369946
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35625/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726009949
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35591/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
otterc commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726249419
@tgravescs @HyukjinKwon I am not entirely sure why these tests would fail with the errors below only for hadoop-2.7 profile.
I see that the `network-yarn` module is build differently. The target jar of this module is a `spark-{version}-yarn-shuffle.jar`. However, that doesn't explain why this will fail only with hadoop-2.7 profile and not for hadoop 3.2. It could also be what @mridulm pointed out that some jar is missing from hadoop-2.7 profile. However, I didn't add any additional dependencies. I didn't create a new `logger` instance. Just using the existing `logger` instance in `YarnShuffleService`.
The reason I am deleting this `common/network-yarn/src/test/java/org/apache/spark/network/yarn/YarnShuffleServiceSuite.java` was that I added this with https://github.com/apache/spark/pull/30062 because I missed that ` resource-managers/yarn/src/test/scala/org/apache/spark/network/yarn/YarnShuffleServiceSuite.scala` existed. I reckon there is no need to have 2 test classes for same class. However, it would have been better to have the test class in the same module.
I can dig deeper into why this happened but that would require more time. In the meantime just wanted to fix the failures with hadoop-2.7 build.
```
[ERROR] testCreateDefaultMergedShuffleFileManagerInstance(org.apache.spark.network.yarn.YarnShuffleServiceSuite) Time elapsed: 0.421 s <<< ERROR!
java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
at org.apache.spark.network.yarn.YarnShuffleServiceSuite.testCreateDefaultMergedShuffleFileManagerInstance(YarnShuffleServiceSuite.java:37)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
at org.apache.spark.network.yarn.YarnShuffleServiceSuite.testCreateDefaultMergedShuffleFileManagerInstance(YarnShuffleServiceSuite.java:37)
[ERROR] testCreateRemoteBlockPushResolverInstance(org.apache.spark.network.yarn.YarnShuffleServiceSuite) Time elapsed: 0 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.network.yarn.YarnShuffleService
at org.apache.spark.network.yarn.YarnShuffleServiceSuite.testCreateRemoteBlockPushResolverInstance(YarnShuffleServiceSuite.java:47)
[ERROR] testInvalidClassNameOfMergeManagerWillUseNoOpInstance(org.apache.spark.network.yarn.YarnShuffleServiceSuite) Time elapsed: 0.001 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.network.yarn.YarnShuffleService
at org.apache.spark.network.yarn.YarnShuffleServiceSuite.testInvalidClassNameOfMergeManagerWillUseNoOpInstance(YarnShuffleServiceSuite.java:57)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-727038664
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726494857
Hi, @otterc , @mridulm , @tgravescs , @HyukjinKwon .
Thank you for making an effort to recover `master` branch quickly.
I'll leave this PR to you guys and hope the recovery occurs soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726856815
failures look unrelated, I'm honestly not sure what they are:
2020-11-13T15:55:21.9268572Z [error] running /home/runner/work/spark/spark/build/sbt -Phadoop-3.2 -Phive-2.3 -Dtest.include.tags=org.apache.spark.tags.ExtendedSQLTest sql/test ; received return code 1
2020-11-13T15:55:21.9487619Z ##[error]Process completed with exit code 18.
2020-11-13T15:55:22.0775315Z ##[group]Run actions/upload-artifact@v2
@dongjoon-hyun have you been this before? Maybe I'm missing something in the logs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726384975
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #30349: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726789691
I'm fine to get something checked in quickly to fix the build but would like to understand it as well. I kicked the build again as failure didn't look related.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30349: [SPARK-32916][SHUFFLE] Remove the newly added YarnShuffleServiceSuite.java
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-725943142
**[Test build #130985 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130985/testReport)** for PR 30349 at commit [`c11f583`](https://github.com/apache/spark/commit/c11f583174d0309637f4daee9c25e2d2a76278e1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org