You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "leosandylh@gmail.com" <le...@gmail.com> on 2014/01/09 14:31:03 UTC
some problems
Hi,
I'm runing a shark sql, but I don't know How spark build these stages and what work it does in each stage .
I waited the job 1 hour , it seems that the task 504 has some problem.
there are some logs in the machine which runs task 504 :
14/01/09 20:22:55 INFO executor.Executor: Finished task ID 287
14/01/09 20:23:00 INFO executor.StandaloneExecutorBackend: Got assigned task 504
14/01/09 20:23:00 INFO executor.Executor: Running task ID 504
14/01/09 20:23:00 INFO executor.Executor: Its epoch is 2
14/01/09 20:23:00 INFO spark.MapOutputTracker: Updating epoch to 2 and clearing cache
14/01/09 20:23:00 INFO spark.MapOutputTracker: Don't have map outputs for shuffle 2, fetching them
14/01/09 20:23:00 INFO spark.MapOutputTracker: Doing the fetch; tracker actor = Actor[akka://spark@OCDC-DD-002:53976/user/MapOutputTracker]
14/01/09 20:23:00 INFO spark.MapOutputTracker: Got the output locations
14/01/09 20:23:00 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight: 50331648, minRequest: 10066329
14/01/09 20:23:00 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight: 50331648, minRequest: 10066329
14/01/09 20:23:00 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight: 50331648, minRequest: 10066329
14/01/09 20:23:00 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight: 50331648, minRequest: 10066329
14/01/09 20:23:00 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 24 non-zero-bytes blocks out of 264 blocks
14/01/09 20:23:00 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 1 remote gets in 74 ms
14/01/09 20:23:00 INFO network.SendingConnection: Initiating connection to [OCDC-DATA-013/172.17.254.33:54930]
14/01/09 20:23:00 INFO network.SendingConnection: Connected to [OCDC-DATA-013/172.17.254.33:54930], 1 messages pending
14/01/09 20:23:00 INFO network.ConnectionManager: Accepted connection from [OCDC-DATA-013/172.17.254.33]
14/01/09 20:23:18 INFO network.SendingConnection: Initiating connection to [OCDC-DATA-012/172.17.254.32:45718]
14/01/09 20:23:18 INFO network.SendingConnection: Connected to [OCDC-DATA-012/172.17.254.32:45718], 1 messages pending
14/01/09 20:23:18 INFO network.ConnectionManager: Accepted connection from [OCDC-DATA-012/172.17.254.32]
14/01/09 20:23:46 INFO network.SendingConnection: Initiating connection to [OCDC-DATA-011/172.17.254.31:51067]
14/01/09 20:23:46 INFO network.SendingConnection: Connected to [OCDC-DATA-011/172.17.254.31:51067], 1 messages pending
14/01/09 20:23:46 INFO network.ConnectionManager: Accepted connection from [OCDC-DATA-011/172.17.254.31]
14/01/09 20:56:48 INFO storage.BlockManager: BlockManager reregistering with master
14/01/09 20:56:48 INFO storage.BlockManagerMaster: Trying to register BlockManager
14/01/09 20:56:48 INFO storage.BlockManagerMaster: Registered BlockManager
14/01/09 20:56:48 INFO storage.BlockManager: Reporting 117 blocks to the master.
14/01/09 21:04:43 INFO storage.BlockManager: BlockManager reregistering with master
14/01/09 21:04:43 INFO storage.BlockManagerMaster: Trying to register BlockManager
14/01/09 21:04:43 INFO storage.BlockManagerMaster: Registered BlockManager
14/01/09 21:04:43 INFO storage.BlockManager: Reporting 117 blocks to the master.
14/01/09 21:06:43 INFO storage.BlockManager: BlockManager reregistering with master
14/01/09 21:06:43 INFO storage.BlockManagerMaster: Trying to register BlockManager
14/01/09 21:06:43 INFO storage.BlockManagerMaster: Registered BlockManager
14/01/09 21:06:43 INFO storage.BlockManager: Reporting 117 blocks to the master.
14/01/09 21:11:48 INFO storage.BlockManager: BlockManager reregistering with master
14/01/09 21:11:48 INFO storage.BlockManagerMaster: Trying to register BlockManager
14/01/09 21:11:48 INFO storage.BlockManagerMaster: Registered BlockManager
14/01/09 21:11:48 INFO storage.BlockManager: Reporting 117 blocks to the master.
14/01/09 21:12:48 INFO storage.BlockManager: BlockManager reregistering with master
14/01/09 21:12:48 INFO storage.BlockManagerMaster: Trying to register BlockManager
14/01/09 21:12:48 INFO storage.BlockManagerMaster: Registered BlockManager
14/01/09 21:12:48 INFO storage.BlockManager: Reporting 117 blocks to the master.
14/01/09 21:13:51 INFO storage.BlockManager: BlockManager reregistering with master
14/01/09 21:13:51 INFO storage.BlockManagerMaster: Trying to register BlockManager
can anyone give me the hint
thank you !
leosandylh@gmail.com