You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2014/09/27 21:44:33 UTC
[jira] [Resolved] (SPARK-1895) Run tests on windows
[ https://issues.apache.org/jira/browse/SPARK-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-1895.
------------------------------
Resolution: Cannot Reproduce
> Run tests on windows
> --------------------
>
> Key: SPARK-1895
> URL: https://issues.apache.org/jira/browse/SPARK-1895
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Windows
> Affects Versions: 0.9.1
> Environment: spark-0.9.1-bin-hadoop1
> Reporter: stribog
> Priority: Trivial
>
> bin\pyspark python\pyspark\rdd.py
> Sometimes tests complete without error >_<.
> Last tests fail log:
> {noformat}
> ....
> 14/05/21 18:31:40 INFO Executor: Running task ID 321
> 14/05/21 18:31:40 INFO Executor: Running task ID 324
> 14/05/21 18:31:40 INFO Executor: Running task ID 322
> 14/05/21 18:31:40 INFO Executor: Running task ID 323
> 14/05/21 18:31:40 INFO PythonRDD: Times: total = 241, boot = 240, init = 1, finish = 0
> 14/05/21 18:31:40 INFO Executor: Serialized size of result for 324 is 607
> 14/05/21 18:31:40 INFO Executor: Sending result for 324 directly to driver
> 14/05/21 18:31:40 INFO Executor: Finished task ID 324
> 14/05/21 18:31:40 INFO TaskSetManager: Finished TID 324 in 248 ms on localhost (progress: 1/4)
> 14/05/21 18:31:40 INFO DAGScheduler: Completed ResultTask(80, 3)
> 14/05/21 18:31:40 INFO PythonRDD: Times: total = 518, boot = 516, init = 2, finish = 0
> 14/05/21 18:31:40 INFO Executor: Serialized size of result for 323 is 607
> 14/05/21 18:31:40 INFO Executor: Sending result for 323 directly to driver
> 14/05/21 18:31:40 INFO Executor: Finished task ID 323
> 14/05/21 18:31:40 INFO TaskSetManager: Finished TID 323 in 528 ms on localhost (progress: 2/4)
> 14/05/21 18:31:40 INFO DAGScheduler: Completed ResultTask(80, 2)
> 14/05/21 18:31:41 INFO PythonRDD: Times: total = 776, boot = 774, init = 2, finish = 0
> 14/05/21 18:31:41 INFO Executor: Serialized size of result for 322 is 607
> 14/05/21 18:31:41 INFO Executor: Sending result for 322 directly to driver
> 14/05/21 18:31:41 INFO Executor: Finished task ID 322
> 14/05/21 18:31:41 INFO TaskSetManager: Finished TID 322 in 785 ms on localhost (progress: 3/4)
> 14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(80, 1)
> 14/05/21 18:31:41 INFO PythonRDD: Times: total = 1043, boot = 1042, init = 1, finish = 0
> 14/05/21 18:31:41 INFO Executor: Serialized size of result for 321 is 607
> 14/05/21 18:31:41 INFO Executor: Sending result for 321 directly to driver
> 14/05/21 18:31:41 INFO Executor: Finished task ID 321
> 14/05/21 18:31:41 INFO TaskSetManager: Finished TID 321 in 1049 ms on localhost (progress: 4/4)
> 14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(80, 0)
> 14/05/21 18:31:41 INFO TaskSchedulerImpl: Removed TaskSet 80.0, whose tasks have all completed, from pool
> 14/05/21 18:31:41 INFO DAGScheduler: Stage 80 (top at <doctest __main__.RDD.top[0]>:1) finished in 1,051 s
> 14/05/21 18:31:41 INFO SparkContext: Job finished: top at <doctest __main__.RDD.top[0]>:1, took 1.053832912 s
> 14/05/21 18:31:41 INFO SparkContext: Starting job: top at <doctest __main__.RDD.top[1]>:1
> 14/05/21 18:31:41 INFO DAGScheduler: Got job 63 (top at <doctest __main__.RDD.top[1]>:1) with 4 output partitions (allowLocal=false)
> 14/05/21 18:31:41 INFO DAGScheduler: Final stage: Stage 81 (top at <doctest __main__.RDD.top[1]>:1)
> 14/05/21 18:31:41 INFO DAGScheduler: Parents of final stage: List()
> 14/05/21 18:31:41 INFO DAGScheduler: Missing parents: List()
> 14/05/21 18:31:41 INFO DAGScheduler: Submitting Stage 81 (PythonRDD[213] at top at <doctest __main__.RDD.top[1]>:1), which has no missing parents
> 14/05/21 18:31:41 INFO DAGScheduler: Submitting 4 missing tasks from Stage 81 (PythonRDD[213] at top at <doctest __main__.RDD.top[1]>:1)
> 14/05/21 18:31:41 INFO TaskSchedulerImpl: Adding task set 81.0 with 4 tasks
> 14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:0 as TID 325 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:0 as 2594 bytes in 0 ms
> 14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:1 as TID 326 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:1 as 2594 bytes in 0 ms
> 14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:2 as TID 327 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:2 as 2594 bytes in 0 ms
> 14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:3 as TID 328 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:3 as 2609 bytes in 1 ms
> 14/05/21 18:31:41 INFO Executor: Running task ID 326
> 14/05/21 18:31:41 INFO Executor: Running task ID 328
> 14/05/21 18:31:41 INFO Executor: Running task ID 327
> 14/05/21 18:31:41 INFO Executor: Running task ID 325
> 14/05/21 18:31:41 INFO CacheManager: Partition rdd_212_3 not found, computing it
> 14/05/21 18:31:41 INFO MemoryStore: ensureFreeSpace(152) called with curMem=1120, maxMem=311387750
> 14/05/21 18:31:41 INFO MemoryStore: Block rdd_212_3 stored as values to memory (estimated size 152.0 B, free 297.0 MB)
> 14/05/21 18:31:41 INFO BlockManagerMasterActor$BlockManagerInfo: Added rdd_212_3 in memory on stribog-pc:37187 (size: 152.0 B, free: 297.0 MB)
> 14/05/21 18:31:41 INFO BlockManagerMaster: Updated info of block rdd_212_3
> 14/05/21 18:31:41 INFO PythonRDD: Times: total = 245, boot = 241, init = 4, finish = 0
> 14/05/21 18:31:41 INFO Executor: Serialized size of result for 328 is 610
> 14/05/21 18:31:41 INFO Executor: Sending result for 328 directly to driver
> 14/05/21 18:31:41 INFO Executor: Finished task ID 328
> 14/05/21 18:31:41 INFO TaskSetManager: Finished TID 328 in 251 ms on localhost (progress: 1/4)
> 14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(81, 3)
> 14/05/21 18:31:41 INFO CacheManager: Partition rdd_212_0 not found, computing it
> 14/05/21 18:31:41 INFO MemoryStore: ensureFreeSpace(128) called with curMem=1272, maxMem=311387750
> 14/05/21 18:31:41 INFO MemoryStore: Block rdd_212_0 stored as values to memory (estimated size 128.0 B, free 297.0 MB)
> 14/05/21 18:31:41 INFO BlockManagerMasterActor$BlockManagerInfo: Added rdd_212_0 in memory on stribog-pc:37187 (size: 128.0 B, free: 297.0 MB)
> 14/05/21 18:31:41 INFO BlockManagerMaster: Updated info of block rdd_212_0
> 14/05/21 18:31:41 INFO PythonRDD: Times: total = 484, boot = 481, init = 3, finish = 0
> 14/05/21 18:31:41 INFO Executor: Serialized size of result for 325 is 607
> 14/05/21 18:31:41 INFO Executor: Sending result for 325 directly to driver
> 14/05/21 18:31:41 INFO Executor: Finished task ID 325
> 14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(81, 0)
> 14/05/21 18:31:41 INFO TaskSetManager: Finished TID 325 in 493 ms on localhost (progress: 2/4)
> 14/05/21 18:31:42 INFO CacheManager: Partition rdd_212_2 not found, computing it
> 14/05/21 18:31:42 INFO MemoryStore: ensureFreeSpace(128) called with curMem=1400, maxMem=311387750
> 14/05/21 18:31:42 INFO MemoryStore: Block rdd_212_2 stored as values to memory (estimated size 128.0 B, free 297.0 MB)
> 14/05/21 18:31:42 INFO BlockManagerMasterActor$BlockManagerInfo: Added rdd_212_2 in memory on stribog-pc:37187 (size: 128.0 B, free: 297.0 MB)
> 14/05/21 18:31:42 INFO BlockManagerMaster: Updated info of block rdd_212_2
> 14/05/21 18:31:42 INFO PythonRDD: Times: total = 728, boot = 724, init = 4, finish = 0
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 327 is 607
> 14/05/21 18:31:42 INFO Executor: Sending result for 327 directly to driver
> 14/05/21 18:31:42 INFO Executor: Finished task ID 327
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 327 in 735 ms on localhost (progress: 3/4)
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(81, 2)
> 14/05/21 18:31:42 INFO CacheManager: Partition rdd_212_1 not found, computing it
> 14/05/21 18:31:42 INFO MemoryStore: ensureFreeSpace(128) called with curMem=1528, maxMem=311387750
> 14/05/21 18:31:42 INFO MemoryStore: Block rdd_212_1 stored as values to memory (estimated size 128.0 B, free 297.0 MB)
> 14/05/21 18:31:42 INFO BlockManagerMasterActor$BlockManagerInfo: Added rdd_212_1 in memory on stribog-pc:37187 (size: 128.0 B, free: 297.0 MB)
> 14/05/21 18:31:42 INFO BlockManagerMaster: Updated info of block rdd_212_1
> 14/05/21 18:31:42 INFO PythonRDD: Times: total = 1487, boot = 1484, init = 3, finish = 0
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 326 is 607
> 14/05/21 18:31:42 INFO Executor: Sending result for 326 directly to driver
> 14/05/21 18:31:42 INFO Executor: Finished task ID 326
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 326 in 1493 ms on localhost (progress: 4/4)
> 14/05/21 18:31:42 INFO TaskSchedulerImpl: Removed TaskSet 81.0, whose tasks have all completed, from pool
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(81, 1)
> 14/05/21 18:31:42 INFO DAGScheduler: Stage 81 (top at <doctest __main__.RDD.top[1]>:1) finished in 1,496 s
> 14/05/21 18:31:42 INFO SparkContext: Job finished: top at <doctest __main__.RDD.top[1]>:1, took 1.498387373 s
> 14/05/21 18:31:42 INFO SparkContext: Starting job: collect at <doctest __main__.RDD.union[1]>:1
> 14/05/21 18:31:42 INFO DAGScheduler: Got job 64 (collect at <doctest __main__.RDD.union[1]>:1) with 8 output partitions (allowLocal=false)
> 14/05/21 18:31:42 INFO DAGScheduler: Final stage: Stage 82 (collect at <doctest __main__.RDD.union[1]>:1)
> 14/05/21 18:31:42 INFO DAGScheduler: Parents of final stage: List()
> 14/05/21 18:31:42 INFO DAGScheduler: Missing parents: List()
> 14/05/21 18:31:42 INFO DAGScheduler: Submitting Stage 82 (UnionRDD[215] at union at null:-2), which has no missing parents
> 14/05/21 18:31:42 INFO DAGScheduler: Submitting 8 missing tasks from Stage 82 (UnionRDD[215] at union at null:-2)
> 14/05/21 18:31:42 INFO TaskSchedulerImpl: Adding task set 82.0 with 8 tasks
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:0 as TID 329 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:0 as 2208 bytes in 0 ms
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:1 as TID 330 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:1 as 2208 bytes in 0 ms
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:2 as TID 331 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:2 as 2208 bytes in 0 ms
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:3 as TID 332 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:3 as 2208 bytes in 0 ms
> 14/05/21 18:31:42 INFO Executor: Running task ID 330
> 14/05/21 18:31:42 INFO Executor: Running task ID 331
> 14/05/21 18:31:42 INFO Executor: Running task ID 332
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 331 is 537
> 14/05/21 18:31:42 INFO Executor: Sending result for 331 directly to driver
> 14/05/21 18:31:42 INFO Executor: Finished task ID 331
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 332 is 537
> 14/05/21 18:31:42 INFO Executor: Running task ID 329
> 14/05/21 18:31:42 INFO Executor: Sending result for 332 directly to driver
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:4 as TID 333 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 330 is 537
> 14/05/21 18:31:42 INFO Executor: Sending result for 330 directly to driver
> 14/05/21 18:31:42 INFO Executor: Finished task ID 330
> 14/05/21 18:31:42 INFO Executor: Finished task ID 332
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:4 as 2208 bytes in 0 ms
> 14/05/21 18:31:42 INFO Executor: Running task ID 333
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 331 in 3 ms on localhost (progress: 1/8)
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 2)
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 332 in 2 ms on localhost (progress: 2/8)
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 329 is 537
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 333 is 537
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 3)
> 14/05/21 18:31:42 INFO Executor: Sending result for 333 directly to driver
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:5 as TID 334 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO Executor: Sending result for 329 directly to driver
> 14/05/21 18:31:42 INFO Executor: Finished task ID 333
> 14/05/21 18:31:42 INFO Executor: Finished task ID 329
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:5 as 2208 bytes in 0 ms
> 14/05/21 18:31:42 INFO Executor: Running task ID 334
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:6 as TID 335 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:6 as 2208 bytes in 0 ms
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:7 as TID 336 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO Executor: Running task ID 335
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 334 is 537
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:7 as 2208 bytes in 0 ms
> 14/05/21 18:31:42 INFO Executor: Sending result for 334 directly to driver
> 14/05/21 18:31:42 INFO Executor: Finished task ID 334
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 1)
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 330 in 5 ms on localhost (progress: 3/8)
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 335 is 537
> 14/05/21 18:31:42 INFO Executor: Sending result for 335 directly to driver
> 14/05/21 18:31:42 INFO Executor: Running task ID 336
> 14/05/21 18:31:42 INFO Executor: Finished task ID 335
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 5)
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 334 in 2 ms on localhost (progress: 4/8)
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 333 in 5 ms on localhost (progress: 5/8)
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 4)
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 336 is 537
> 14/05/21 18:31:42 INFO Executor: Sending result for 336 directly to driver
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 335 in 3 ms on localhost (progress: 6/8)
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 6)
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 329 in 7 ms on localhost (progress: 7/8)
> 14/05/21 18:31:42 INFO Executor: Finished task ID 336
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 336 in 3 ms on localhost (progress: 8/8)
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 0)
> 14/05/21 18:31:42 INFO TaskSchedulerImpl: Removed TaskSet 82.0, whose tasks have all completed, from pool
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 7)
> 14/05/21 18:31:42 INFO DAGScheduler: Stage 82 (collect at <doctest __main__.RDD.union[1]>:1) finished in 0,008 s
> 14/05/21 18:31:42 INFO SparkContext: Job finished: collect at <doctest __main__.RDD.union[1]>:1, took 0.010936769 s
> 14/05/21 18:31:42 INFO SparkContext: Starting job: variance at <doctest __main__.RDD.variance[0]>:1
> 14/05/21 18:31:42 INFO DAGScheduler: Got job 65 (variance at <doctest __main__.RDD.variance[0]>:1) with 4 output partitions (allowLocal=false)
> 14/05/21 18:31:42 INFO DAGScheduler: Final stage: Stage 83 (variance at <doctest __main__.RDD.variance[0]>:1)
> 14/05/21 18:31:42 INFO DAGScheduler: Parents of final stage: List()
> 14/05/21 18:31:42 INFO DAGScheduler: Missing parents: List()
> 14/05/21 18:31:42 INFO DAGScheduler: Submitting Stage 83 (PythonRDD[217] at variance at <doctest __main__.RDD.variance[0]>:1), which has no missing parents
> 14/05/21 18:31:42 INFO DAGScheduler: Submitting 4 missing tasks from Stage 83 (PythonRDD[217] at variance at <doctest __main__.RDD.variance[0]>:1)
> 14/05/21 18:31:42 INFO TaskSchedulerImpl: Adding task set 83.0 with 4 tasks
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:0 as TID 337 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:0 as 2486 bytes in 0 ms
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:1 as TID 338 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:1 as 2514 bytes in 0 ms
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:2 as TID 339 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:2 as 2514 bytes in 0 ms
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:3 as TID 340 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:3 as 2514 bytes in 0 ms
> 14/05/21 18:31:42 INFO Executor: Running task ID 338
> 14/05/21 18:31:42 INFO Executor: Running task ID 340
> 14/05/21 18:31:42 INFO Executor: Running task ID 339
> 14/05/21 18:31:42 INFO Executor: Running task ID 337
> 14/05/21 18:31:43 INFO PythonRDD: Times: total = 265, boot = 264, init = 1, finish = 0
> 14/05/21 18:31:43 INFO Executor: Serialized size of result for 340 is 682
> 14/05/21 18:31:43 INFO Executor: Sending result for 340 directly to driver
> 14/05/21 18:31:43 INFO Executor: Finished task ID 340
> 14/05/21 18:31:43 INFO TaskSetManager: Finished TID 340 in 278 ms on localhost (progress: 1/4)
> 14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 3)
> 14/05/21 18:31:43 INFO PythonRDD: Times: total = 519, boot = 517, init = 2, finish = 0
> 14/05/21 18:31:43 INFO Executor: Serialized size of result for 337 is 681
> 14/05/21 18:31:43 INFO Executor: Sending result for 337 directly to driver
> 14/05/21 18:31:43 INFO Executor: Finished task ID 337
> 14/05/21 18:31:43 INFO TaskSetManager: Finished TID 337 in 527 ms on localhost (progress: 2/4)
> 14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 0)
> 14/05/21 18:31:43 INFO PythonRDD: Times: total = 779, boot = 778, init = 1, finish = 0
> 14/05/21 18:31:43 INFO Executor: Serialized size of result for 338 is 682
> 14/05/21 18:31:43 INFO Executor: Sending result for 338 directly to driver
> 14/05/21 18:31:43 INFO Executor: Finished task ID 338
> 14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 1)
> 14/05/21 18:31:43 INFO TaskSetManager: Finished TID 338 in 786 ms on localhost (progress: 3/4)
> 14/05/21 18:31:43 INFO PythonRDD: Times: total = 1025, boot = 1023, init = 1, finish = 1
> 14/05/21 18:31:43 INFO Executor: Serialized size of result for 339 is 682
> 14/05/21 18:31:43 INFO Executor: Sending result for 339 directly to driver
> 14/05/21 18:31:43 INFO Executor: Finished task ID 339
> 14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 2)
> 14/05/21 18:31:43 INFO TaskSetManager: Finished TID 339 in 1030 ms on localhost (progress: 4/4)
> 14/05/21 18:31:43 INFO TaskSchedulerImpl: Removed TaskSet 83.0, whose tasks have all completed, from pool
> 14/05/21 18:31:43 INFO DAGScheduler: Stage 83 (variance at <doctest __main__.RDD.variance[0]>:1) finished in 1,032 s
> 14/05/21 18:31:43 INFO SparkContext: Job finished: variance at <doctest __main__.RDD.variance[0]>:1, took 1.034616593 s
> 14/05/21 18:31:44 INFO SparkContext: Starting job: collect at <doctest __main__.RDD.zip[2]>:1
> 14/05/21 18:31:44 INFO DAGScheduler: Got job 66 (collect at <doctest __main__.RDD.zip[2]>:1) with 4 output partitions (allowLocal=false)
> 14/05/21 18:31:44 INFO DAGScheduler: Final stage: Stage 84 (collect at <doctest __main__.RDD.zip[2]>:1)
> 14/05/21 18:31:44 INFO DAGScheduler: Parents of final stage: List()
> 14/05/21 18:31:44 INFO DAGScheduler: Missing parents: List()
> 14/05/21 18:31:44 INFO DAGScheduler: Submitting Stage 84 (ZippedRDD[220] at zip at null:-2), which has no missing parents
> 14/05/21 18:31:44 INFO DAGScheduler: Submitting 4 missing tasks from Stage 84 (ZippedRDD[220] at zip at null:-2)
> 14/05/21 18:31:44 INFO TaskSchedulerImpl: Adding task set 84.0 with 4 tasks
> 14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:0 as TID 341 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:0 as 1581 bytes in 1 ms
> 14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:1 as TID 342 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:1 as 1581 bytes in 0 ms
> 14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:2 as TID 343 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:2 as 1581 bytes in 0 ms
> 14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:3 as TID 344 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:3 as 1612 bytes in 0 ms
> 14/05/21 18:31:44 INFO Executor: Running task ID 341
> 14/05/21 18:31:44 INFO Executor: Running task ID 343
> 14/05/21 18:31:44 INFO Executor: Running task ID 344
> 14/05/21 18:31:44 INFO Executor: Running task ID 342
> 14/05/21 18:31:44 INFO Executor: Serialized size of result for 343 is 630
> 14/05/21 18:31:44 INFO Executor: Serialized size of result for 344 is 667
> 14/05/21 18:31:44 INFO Executor: Serialized size of result for 341 is 630
> 14/05/21 18:31:44 INFO Executor: Sending result for 344 directly to driver
> 14/05/21 18:31:44 INFO Executor: Sending result for 343 directly to driver
> 14/05/21 18:31:44 INFO Executor: Finished task ID 343
> 14/05/21 18:31:44 INFO Executor: Finished task ID 344
> 14/05/21 18:31:44 INFO Executor: Sending result for 341 directly to driver
> 14/05/21 18:31:44 INFO Executor: Serialized size of result for 342 is 630
> 14/05/21 18:31:44 INFO TaskSetManager: Finished TID 344 in 3 ms on localhost (progress: 1/4)
> 14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 3)
> 14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 2)
> 14/05/21 18:31:44 INFO Executor: Finished task ID 341
> 14/05/21 18:31:44 INFO TaskSetManager: Finished TID 343 in 4 ms on localhost (progress: 2/4)
> 14/05/21 18:31:44 INFO Executor: Sending result for 342 directly to driver
> 14/05/21 18:31:44 INFO Executor: Finished task ID 342
> 14/05/21 18:31:44 INFO TaskSetManager: Finished TID 341 in 5 ms on localhost (progress: 3/4)
> 14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 0)
> 14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 1)
> 14/05/21 18:31:44 INFO TaskSetManager: Finished TID 342 in 5 ms on localhost (progress: 4/4)
> 14/05/21 18:31:44 INFO DAGScheduler: Stage 84 (collect at <doctest __main__.RDD.zip[2]>:1) finished in 0,006 s
> 14/05/21 18:31:44 INFO TaskSchedulerImpl: Removed TaskSet 84.0, whose tasks have all completed, from pool
> 14/05/21 18:31:44 INFO SparkContext: Job finished: collect at <doctest __main__.RDD.zip[2]>:1, took 0.010312512 s
> 14/05/21 18:31:45 INFO MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
> 14/05/21 18:31:45 INFO ConnectionManager: Selector thread was interrupted!
> 14/05/21 18:31:45 INFO ConnectionManager: ConnectionManager stopped
> 14/05/21 18:31:45 INFO MemoryStore: MemoryStore cleared
> 14/05/21 18:31:45 INFO BlockManager: BlockManager stopped
> 14/05/21 18:31:45 INFO BlockManagerMasterActor: Stopping BlockManagerMaster
> 14/05/21 18:31:45 INFO BlockManagerMaster: BlockManagerMaster stopped
> 14/05/21 18:31:45 INFO SparkContext: Successfully stopped SparkContext
> 14/05/21 18:31:45 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
> 14/05/21 18:31:45 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
> 14/05/21 18:31:45 INFO Remoting: Remoting shut down
> 14/05/21 18:31:45 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
> Exception in thread "delete Spark temp dir C:\Temp\spark-b8e5c57a-dad1-481b-a20e-cd0e6a2ced75"
> java.io.IOException: Failed to delete: C:\Temp\spark-b8e5c57a-dad1-481b-a20e-cd0e6
> a2ced75\tmpbdykfu
> at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:483)
> at org.apache.spark.util.Utils$$anonfun$deleteRecursively$1.apply(Utils.scala:479)
> at org.apache.spark.util.Utils$$anonfun$deleteRecursively$1.apply(Utils.scala:478)
> at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
> at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:478)
> at org.apache.spark.util.Utils$$anon$4.run(Utils.scala:212)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org