You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by rapelly kartheek <ka...@gmail.com> on 2014/11/27 07:17:15 UTC
How the sequence of blockManagerId's are constructed in spark/*/storage/blockManagerMasterActor.getPeers()?
Hi,
I've been fiddling with spark/*/storage/blockManagerMasterActor.getPeers()
definition in the context of blockManagerMaster.askDriverWithReply()
sending a request GetPeers().
1) I couldn't understand what the 'selfIndex' is used for?.
2) Also, I tried modifying the 'peers' array by just eliminating some
blockManagerId's and passed the modified one to the tabulate method. The
application gets executed, but I find that the
blockManagerMaster.askDriverWithReply() recieves the sequence of
blockManagerIds that include the ones I have eliminated previously.
For example,
My original 'peers' array contained 5 blockManagerId's: BlockManagerId(2,
s2, 39997, 0), BlockManagerId(1, s4, 35874, 0),BlockManagerId(3, s1, 33738,
0), BlockManagerId(0, s3, 38207, 0), BlockManagerId(<driver>, karthik,
34388, 0).
I modified it to peers1 having 3 blockManagerId's : BlockManagerId(2, s2,
39997, 0), BlockManagerId(1, s4, 35874, 0), BlockManagerId(3, s1, 33738, 0).
Then I passed this modified peers1 array for the sequence conversion:
'Array.tabulate[BlockManagerId](size) { i => peers1((selfIndex + i + 1) %
peers1.length) }.toSeq"
But, finally when the /storage/blockManagerMaster.askDriverWithReply() gets
the result, it contains the blockManagerIds that I have eliminated
purposely.
Can someone please make me understand how this seq[BlockManagerId] is
constructed?
Thank you!
Fwd: How the sequence of blockManagerId's are constructed in spark/*/storage/blockManagerMasterActor.getPeers()?
Posted by rapelly kartheek <ka...@gmail.com>.
---------- Forwarded message ----------
From: rapelly kartheek <ka...@gmail.com>
Date: Thu, Nov 27, 2014 at 11:47 AM
Subject: How the sequence of blockManagerId's are constructed in
spark/*/storage/blockManagerMasterActor.getPeers()?
To: user@spark.apache.org
Hi,
I've been fiddling with spark/*/storage/blockManagerMasterActor.getPeers()
definition in the context of blockManagerMaster.askDriverWithReply()
sending a request GetPeers().
1) I couldn't understand what the 'selfIndex' is used for?.
2) Also, I tried modifying the 'peers' array by just eliminating some
blockManagerId's and passed the modified one to the tabulate method. The
application gets executed, but I find that the
blockManagerMaster.askDriverWithReply() recieves the sequence of
blockManagerIds that include the ones I have eliminated previously.
For example,
My original 'peers' array contained 5 blockManagerId's: BlockManagerId(2,
s2, 39997, 0), BlockManagerId(1, s4, 35874, 0),BlockManagerId(3, s1, 33738,
0), BlockManagerId(0, s3, 38207, 0), BlockManagerId(<driver>, karthik,
34388, 0).
I modified it to peers1 having 3 blockManagerId's : BlockManagerId(2, s2,
39997, 0), BlockManagerId(1, s4, 35874, 0), BlockManagerId(3, s1, 33738, 0).
Then I passed this modified peers1 array for the sequence conversion:
'Array.tabulate[BlockManagerId](size) { i => peers1((selfIndex + i + 1) %
peers1.length) }.toSeq"
But, finally when the /storage/blockManagerMaster.askDriverWithReply() gets
the result, it contains the blockManagerIds that I have eliminated
purposely.
Can someone please make me understand how this seq[BlockManagerId] is
constructed?
Thank you!