You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/04/11 06:10:53 UTC

[GitHub] [spark] LuciferYang opened a new pull request, #36139: [SPARK-38678][TESTS][FOLLOWUP] Enable RocksDB tests in `AppStatusStoreSuite` and `StreamingQueryStatusListenerSuite` on Apple Silicon on MacOS

LuciferYang opened a new pull request, #36139:
URL: https://github.com/apache/spark/pull/36139

   ### What changes were proposed in this pull request?
   This pr aims to enable RocksDB tests in `AppStatusStoreSuite` and `StreamingQueryStatusListenerSuite` on Apple Silicon on MacOS, it's a followup of SPARK-38678.
   
   
   ### Why are the changes needed?
   Enable more RocksDB related test on Apple Silicon on MacOS
   
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   
   ### How was this patch tested?
   Pass GA


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] LuciferYang closed pull request #36139: [SPARK-38678][TESTS][FOLLOWUP] Enable RocksDB tests in `AppStatusStoreSuite` and `StreamingQueryStatusListenerSuite` on Apple Silicon on MacOS

Posted by GitBox <gi...@apache.org>.
LuciferYang closed pull request #36139: [SPARK-38678][TESTS][FOLLOWUP] Enable RocksDB tests in `AppStatusStoreSuite` and `StreamingQueryStatusListenerSuite` on Apple Silicon on MacOS
URL: https://github.com/apache/spark/pull/36139


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] LuciferYang commented on a diff in pull request #36139: [SPARK-38678][TESTS][FOLLOWUP] Enable RocksDB tests in `AppStatusStoreSuite` and `StreamingQueryStatusListenerSuite` on Apple Silicon on MacOS

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #36139:
URL: https://github.com/apache/spark/pull/36139#discussion_r847017711


##########
core/src/test/scala/org/apache/spark/status/AppStatusStoreSuite.scala:
##########
@@ -106,12 +102,22 @@ class AppStatusStoreSuite extends SparkFunSuite {
     new AppStatusStore(store)
   }
 
-  Seq(
-    "disk leveldb" -> createAppStore(disk = true, HybridStoreDiskBackend.LEVELDB, live = false),
-    "disk rocksdb" -> createAppStore(disk = true, HybridStoreDiskBackend.ROCKSDB, live = false),
-    "in memory" -> createAppStore(disk = false, live = false),
-    "in memory live" -> createAppStore(disk = false, live = true)
-  ).foreach { case (hint, appStore) =>
+  private val cases = {
+    val baseCases = Seq(
+      "disk rocksdb" -> createAppStore(disk = true, HybridStoreDiskBackend.ROCKSDB, live = false),
+      "in memory" -> createAppStore(disk = false, live = false),
+      "in memory live" -> createAppStore(disk = false, live = true)
+    )
+    if(Utils.isMacOnAppleSilicon) {

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun closed pull request #36139: [SPARK-38678][TESTS][FOLLOWUP] Enable RocksDB tests in `AppStatusStoreSuite` and `StreamingQueryStatusListenerSuite` on Apple Silicon on MacOS

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #36139: [SPARK-38678][TESTS][FOLLOWUP] Enable RocksDB tests in `AppStatusStoreSuite` and `StreamingQueryStatusListenerSuite` on Apple Silicon on MacOS
URL: https://github.com/apache/spark/pull/36139


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #36139: [SPARK-38678][TESTS][FOLLOWUP] Enable RocksDB tests in `AppStatusStoreSuite` and `StreamingQueryStatusListenerSuite` on Apple Silicon on MacOS

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #36139:
URL: https://github.com/apache/spark/pull/36139#issuecomment-1098321362

   Merged to master. Thank you, @LuciferYang .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #36139: [SPARK-38678][TESTS][FOLLOWUP] Enable RocksDB tests in `AppStatusStoreSuite` and `StreamingQueryStatusListenerSuite` on Apple Silicon on MacOS

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #36139:
URL: https://github.com/apache/spark/pull/36139#issuecomment-1095474666

   Ya, indeed. GitHub Action seems to be fail due to its own flakiness.
   ```
   Error: Unhandled error: Error: There was a new unsynced commit pushed. Please retrigger the workflow.
   [124](https://github.com/apache/spark/runs/5972998078?check_suite_focus=true#step:2:124)
       at async main (/home/runner/work/_actions/actions/github-script/f05a81df23035049204b043b50c3322045ce7eb3/dist/index.js:6147:20)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] LuciferYang commented on pull request #36139: [SPARK-38678][TESTS][FOLLOWUP] Enable RocksDB tests in `AppStatusStoreSuite` and `StreamingQueryStatusListenerSuite` on Apple Silicon on MacOS

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #36139:
URL: https://github.com/apache/spark/pull/36139#issuecomment-1095007336

   Looks like GA is faulty, retry several times with the following error:
   
   <img width="1026" alt="image" src="https://user-images.githubusercontent.com/1475305/162742345-33282b54-42c6-4d08-bcb8-b8d06ac51b30.png">
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] LuciferYang commented on pull request #36139: [SPARK-38678][TESTS][FOLLOWUP] Enable RocksDB tests in `AppStatusStoreSuite` and `StreamingQueryStatusListenerSuite` on Apple Silicon on MacOS

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #36139:
URL: https://github.com/apache/spark/pull/36139#issuecomment-1097571842

   https://github.com/LuciferYang/spark/actions/runs/2158563267
   
   <img width="1208" alt="image" src="https://user-images.githubusercontent.com/1475305/163106093-949cbafb-ccdf-4ec6-b8de-d0de916612a9.png">
   
   All test passed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #36139: [SPARK-38678][TESTS][FOLLOWUP] Enable RocksDB tests in `AppStatusStoreSuite` and `StreamingQueryStatusListenerSuite` on Apple Silicon on MacOS

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on code in PR #36139:
URL: https://github.com/apache/spark/pull/36139#discussion_r847015815


##########
core/src/test/scala/org/apache/spark/status/AppStatusStoreSuite.scala:
##########
@@ -106,12 +102,22 @@ class AppStatusStoreSuite extends SparkFunSuite {
     new AppStatusStore(store)
   }
 
-  Seq(
-    "disk leveldb" -> createAppStore(disk = true, HybridStoreDiskBackend.LEVELDB, live = false),
-    "disk rocksdb" -> createAppStore(disk = true, HybridStoreDiskBackend.ROCKSDB, live = false),
-    "in memory" -> createAppStore(disk = false, live = false),
-    "in memory live" -> createAppStore(disk = false, live = true)
-  ).foreach { case (hint, appStore) =>
+  private val cases = {
+    val baseCases = Seq(
+      "disk rocksdb" -> createAppStore(disk = true, HybridStoreDiskBackend.ROCKSDB, live = false),
+      "in memory" -> createAppStore(disk = false, live = false),
+      "in memory live" -> createAppStore(disk = false, live = true)
+    )
+    if(Utils.isMacOnAppleSilicon) {

Review Comment:
   nit. space.
   ```scala
   - if(Utils.isMacOnAppleSilicon) {
   + if (Utils.isMacOnAppleSilicon) {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] LuciferYang commented on pull request #36139: [SPARK-38678][TESTS][FOLLOWUP] Enable RocksDB tests in `AppStatusStoreSuite` and `StreamingQueryStatusListenerSuite` on Apple Silicon on MacOS

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #36139:
URL: https://github.com/apache/spark/pull/36139#issuecomment-1098638984

   thanks @dongjoon-hyun 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org