[GitHub] [spark] dongjoon-hyun commented on pull request #41391: [SPARK-43888][CORE] Relocate Logging to common/utils

["dongjoon-hyun (via GitHub)" <gi...@apache.org>]

dongjoon-hyun commented on PR #41391:
URL: https://github.com/apache/spark/pull/41391#issuecomment-1570594827

   I manually verified the failed test suite.
   
   ```
   [info] TaskSetManagerSuite:
   [info] - TaskSet with no preferences (1 second, 477 milliseconds)
   [info] - multiple offers with no preferences (151 milliseconds)
   [info] - skip unsatisfiable locality levels (79 milliseconds)
   [info] - basic delay scheduling (89 milliseconds)
   [info] - we do not need to delay scheduling when we only have noPref tasks in the queue (123 milliseconds)
   [info] - delay scheduling with fallback (1 second, 371 milliseconds)
   [info] - delay scheduling with failed hosts (156 milliseconds)
   [info] - task result lost (163 milliseconds)
   [info] - repeated failures lead to task set abortion (127 milliseconds)
   [info] - executors should be excluded after task failure, in spite of locality preferences (264 milliseconds)
   [info] - new executors get added and lost (173 milliseconds)
   [info] - Executors exit for reason unrelated to currently running tasks (136 milliseconds)
   [info] - SPARK-31837: Shift to the new highest locality level if there is when recomputeLocality (109 milliseconds
   [info] - SPARK-41469: task doesn't need to rerun on executor lost if shuffle data has migrated (279 milliseconds)
   [info] - SPARK-32653: Decommissioned host should not be used to calculate locality levels (129 milliseconds)
   [info] - SPARK-32653: Decommissioned executor should not be used to calculate locality levels (113 milliseconds)
   [info] - test RACK_LOCAL tasks (199 milliseconds)
   [info] - do not emit warning when serialized task is small (96 milliseconds)
   [info] - emit warning when serialized task is large (166 milliseconds)
   [info] - Not serializable exception thrown if the task cannot be serialized (99 milliseconds)
   [info] - abort the job if total size of results is too large (2 seconds, 573 milliseconds)
   [info] - SPARK-32470: do not check total size of intermediate stages (9 seconds, 888 milliseconds)
   [info] - [SPARK-13931] taskSetManager should not send Resubmitted tasks after being a zombie (175 milliseconds)
   [info] - [SPARK-22074] Task killed by other attempt task should not be resubmitted (151 milliseconds)
   [info] - speculative and noPref task should be scheduled after node-local (79 milliseconds)
   [info] - node-local tasks should be scheduled right away when there are only node-local and no-preference tasks (47 milliseconds)
   [info] - SPARK-4939: node-local tasks should be scheduled right after process-local tasks finished (45 milliseconds)
   [info] - SPARK-4939: no-pref tasks should be scheduled after process-local tasks finished (111 milliseconds)
   [info] - Ensure TaskSetManager is usable after addition of levels (141 milliseconds)
   [info] - Test that locations with HDFSCacheTaskLocation are treated as PROCESS_LOCAL. (113 milliseconds)
   [info] - Test TaskLocation for different host type. (6 milliseconds)
   [info] - Kill other task attempts when one attempt belonging to the same task succeeds (63 milliseconds)
   [info] - Killing speculative tasks does not count towards aborting the taskset (58 milliseconds)
   [info] - SPARK-19868: DagScheduler only notified of taskEnd when state is ready (555 milliseconds)
   [info] - SPARK-17894: Verify TaskSetManagers for different stage attempts have unique names (73 milliseconds)
   [info] - don't update excludelist for shuffle-fetch failures, preemption, denied commits, or killed tasks (80 milliseconds)
   [info] - update application healthTracker for shuffle-fetch (131 milliseconds)
   [info] - update healthTracker before adding pending task to avoid race condition (54 milliseconds)
   [info] - SPARK-21563 context's added jars shouldn't change mid-TaskSet (43 milliseconds)
   [info] - SPARK-24677: Avoid NoSuchElementException from MedianHeap (65 milliseconds)
   [info] - SPARK-24755 Executor loss can cause task to not be resubmitted (125 milliseconds)
   [info] - SPARK-13343 speculative tasks that didn't commit shouldn't be marked as success (64 milliseconds)
   [info] - SPARK-13704 Rack Resolution is done with a batch of de-duped hosts (75 milliseconds)
   [info] - TaskSetManager passes task resource along (52 milliseconds)
   [info] - SPARK-26755 Ensure that a speculative task is submitted only once for execution (106 milliseconds)
   [info] - SPARK-26755 Ensure that a speculative task obeys original locality preferences (72 milliseconds)
   [info] - SPARK-29976 when a speculation time threshold is provided, should speculative run the task even if there are not enough successful runs, total tasks: 1 (66 milliseconds)
   [info] - SPARK-29976: when the speculation time threshold is not provided,don't speculative run if there are not enough successful runs, total tasks: 1 (85 milliseconds)
   [info] - SPARK-29976 when a speculation time threshold is provided, should speculative run the task even if there are not enough successful runs, total tasks: 2 (43 milliseconds)
   [info] - SPARK-29976: when the speculation time threshold is not provided,don't speculative run if there are not enough successful runs, total tasks: 2 (35 milliseconds)
   [info] - SPARK-29976 when a speculation time threshold is provided, should not speculative if there are too many tasks in the stage even though time threshold is provided (37 milliseconds)
   [info] - SPARK-21040: Check speculative tasks are launched when an executor is decommissioned and the tasks running on it cannot finish within EXECUTOR_DECOMMISSION_KILL_INTERVAL (45 milliseconds)
   [info] - SPARK-29976 Regular speculation configs should still take effect even when a threshold is provided (31 milliseconds)
   [info] - SPARK-30417 when spark.task.cpus is greater than spark.executor.cores due to standalone settings, speculate if there is only one task in the stage (51 milliseconds)
   [info] - TaskOutputFileAlreadyExistException lead to task set abortion (35 milliseconds)
   [info] - SPARK-30359: don't clean executorsPendingToRemove at the beginning of CoarseGrainedSchedulerBackend.reset (4 seconds, 523 milliseconds)
   java.util.concurrent.RejectedExecutionException: Task Future(<not completed>) rejected from java.util.concurrent.ThreadPoolExecutor@34c528e5[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0]
   	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
   	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
   	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
   	at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:21)
   	at scala.concurrent.impl.Promise$Transformation.submitWithValue(Promise.scala:429)
   	at scala.concurrent.impl.Promise$DefaultPromise.submitWithValue(Promise.scala:338)
   	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete0(Promise.scala:285)
   	at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:504)
   	at scala.concurrent.ExecutionContext$parasitic$.execute(ExecutionContext.scala:222)
   	at scala.concurrent.impl.Promise$Transformation.submitWithValue(Promise.scala:429)
   	at scala.concurrent.impl.Promise$DefaultPromise.submitWithValue(Promise.scala:338)
   	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete0(Promise.scala:285)
   	at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:504)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   	at java.lang.Thread.run(Thread.java:748)
   [info] - SPARK-33741 Test minimum amount of time a task runs before being considered for speculation (142 milliseconds)
   [info] - SPARK-32170: test speculation for TaskSet with single task (50 milliseconds)
   [info] - SPARK-32170: test SPECULATION_MIN_THRESHOLD for speculating inefficient tasks (124 milliseconds)
   [info] - SPARK-32170: test SPECULATION_EFFICIENCY_TASK_PROCESS_RATE_MULTIPLIER for speculating inefficient tasks (182 milliseconds)
   [info] - SPARK-32170: test SPECULATION_EFFICIENCY_TASK_DURATION_FACTOR for speculating tasks (160 milliseconds)
   [info] - SPARK-37580: Reset numFailures when one of task attempts succeeds (59 milliseconds)
   [info] - SPARK-40094: Send TaskEnd if task failed with NotSerializableException or TaskOutputFileAlreadyExistException (1 second, 511 milliseconds)
   [info] Run completed in 33 seconds, 331 milliseconds.
   [info] Total number of tests run: 63
   [info] Suites: completed 1, aborted 0
   [info] Tests: succeeded 63, failed 0, canceled 0, ignored 0, pending 0
   [info] All tests passed.
   [success] Total time: 230 s (03:50), completed May 31, 2023 10:01:01 AM
   ```
   
   Merged to master. Thank you, @amaliujia and all!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org