You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/11 05:19:41 UTC
[GitHub] [spark] dongjoon-hyun opened a new pull request #34548: [SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon
dongjoon-hyun opened a new pull request #34548:
URL: https://github.com/apache/spark/pull/34548
### What changes were proposed in this pull request?
This PR aims to add `ExtendedLevelDBTest ` to disable `LevelDB` selectively on Apple Silicon.
- Add `ExtendedLevelDBTest ` test tag
- Disable LevelDB test cases from `kvstore` module/`HistoryServerSuite`/`AppStatusStoreSuite`.
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
No. This is a test-only PR.
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34548: [SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34548:
URL: https://github.com/apache/spark/pull/34548#issuecomment-966029140
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49552/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34548: [SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34548:
URL: https://github.com/apache/spark/pull/34548#issuecomment-966059287
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49552/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34548: [SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34548:
URL: https://github.com/apache/spark/pull/34548#issuecomment-965999081
**[Test build #145083 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145083/testReport)** for PR 34548 at commit [`3c3605e`](https://github.com/apache/spark/commit/3c3605e063dd1601ce434d7a80b4aee38fe77fb6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34548: [SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34548:
URL: https://github.com/apache/spark/pull/34548#issuecomment-966042392
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #34548: [SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #34548:
URL: https://github.com/apache/spark/pull/34548
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #34548: [SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34548:
URL: https://github.com/apache/spark/pull/34548#issuecomment-972626941
> SGTM
Let me do it later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #34548: [SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #34548:
URL: https://github.com/apache/spark/pull/34548#issuecomment-972626941
> SGTM
Let me do this later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34548: [SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34548:
URL: https://github.com/apache/spark/pull/34548#issuecomment-966072518
**[Test build #145083 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145083/testReport)** for PR 34548 at commit [`3c3605e`](https://github.com/apache/spark/commit/3c3605e063dd1601ce434d7a80b4aee38fe77fb6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34548: [SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34548:
URL: https://github.com/apache/spark/pull/34548#issuecomment-966006657
Thank you, @HyukjinKwon .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34548: [SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34548:
URL: https://github.com/apache/spark/pull/34548#issuecomment-966075663
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34548: [SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34548:
URL: https://github.com/apache/spark/pull/34548#issuecomment-972572642
SGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34548: [SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34548:
URL: https://github.com/apache/spark/pull/34548#discussion_r747216390
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingSessionWindowSuite.scala
##########
@@ -50,8 +51,7 @@ class StreamingSessionWindowSuite extends StreamTest
(SQLConf.STATE_STORE_PROVIDER_CLASS.key, value.stripSuffix("$"))
}
// RocksDB doesn't support Apple Silicon yet
- if (System.getProperty("os.name").equals("Mac OS X") &&
- System.getProperty("os.arch").equals("aarch64")) {
+ if (Utils.isMac && System.getProperty("os.arch").equals("aarch64")) {
Review comment:
I piggy-back this replacement from `System.getProperty("os.name").equals("Mac OS X")` to `Utils.isMac`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34548: [SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34548:
URL: https://github.com/apache/spark/pull/34548#issuecomment-966075664
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34548: [SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34548:
URL: https://github.com/apache/spark/pull/34548#discussion_r747209410
##########
File path: common/kvstore/src/test/java/org/apache/spark/util/kvstore/LevelDBIteratorSuite.java
##########
@@ -39,6 +41,7 @@ public static void cleanup() throws Exception {
@Override
protected KVStore createStore() throws Exception {
+ assumeFalse(SystemUtils.IS_OS_MAC_OSX && System.getProperty("os.arch").equals("aarch64"));
Review comment:
We cannot use `ExtendedLevelDBTest` in Java test.
##########
File path: common/kvstore/src/test/java/org/apache/spark/util/kvstore/LevelDBSuite.java
##########
@@ -50,6 +52,7 @@ public void cleanup() throws Exception {
@Before
public void setup() throws Exception {
+ assumeFalse(SystemUtils.IS_OS_MAC_OSX && System.getProperty("os.arch").equals("aarch64"));
Review comment:
We cannot use `ExtendedLevelDBTest` in Java test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34548: [SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34548:
URL: https://github.com/apache/spark/pull/34548#discussion_r747209822
##########
File path: core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
##########
@@ -1651,6 +1651,8 @@ class FsHistoryProviderSuite extends SparkFunSuite with Matchers with Logging {
.set(FAST_IN_PROGRESS_PARSING, true)
if (!inMemory) {
+ // LevelDB doesn't support Apple Silicon yet
+ assume(!(Utils.isMac && System.getProperty("os.arch").equals("aarch64")))
Review comment:
This test case will run `in-memory` test still on M1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34548: [SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34548:
URL: https://github.com/apache/spark/pull/34548#issuecomment-965999081
**[Test build #145083 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145083/testReport)** for PR 34548 at commit [`3c3605e`](https://github.com/apache/spark/commit/3c3605e063dd1601ce434d7a80b4aee38fe77fb6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #34548: [SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34548:
URL: https://github.com/apache/spark/pull/34548#issuecomment-972567773
Should we extract `SystemUtils.IS_OS_MAC_OSX && SystemUtils.OS_ARCH.equals("aarch64")` into an independent `val`, such as `Utils.isAppleSilicon` @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org