"Local-only" aggregators

[Alessio Arleo <in...@icloud.com>]

Hello everybody

I was wondering if it was possible to extend the concept of aggregator from a “global” to a “local-only” perspective. 

Normally, aggregators DO cause network traffic because of the cycle: Workers -> Aggregator Owner-> MasterAggregator -> AggregatorOwner -> Workers

What if I’d like to fetch and aggregate values as I would normally do with aggregators but without causing this traffic? Let’s assume this situation:

1 - Define a custom partitioning class and let it partition the graph. This is the partition used to assign vertices to workers. 
2 - in the computation class, every time che compute method is called on a vertex, the data needed for computation is stored inside the vertex neighbours but also in non-neighbouring vertices (think about Force Directed layout algorithm for example; to compute the forces, is necessary the distance between neighbouring and not-neighbouring vertices, applying different kind of forces).
	
— Given that the compute class is computing on vertex X
	a - I pick information from X neighbours as I would normally do (iterating its edges or the incoming messages)
	b - When it comes to non-neighbouring vertices I would like to use data from X worker only.

The first thing I tried to understand before asking this question was: does this make any sense? I am probably wrong, but this actually does. If I partition my graph to maximize locality, what I am actually trying to do is to reduce the network traffic as much as possibile. 

My doubt is that if I use aggregators to achieve the result the network traffic would be heavy, probably losing the advantages of the initial partitioning. What if I could access and modify an aggregator-like local data structure in the same fashion (i.e. “getAggregatedValue”) but without broadcasting it (assuming that I do not need the aggregator to be accessible to every worker)? Or could it be possibile to manually assign partition owners in order to minimise network traffic (if I need to aggregate all values from vertices in partition 3 and 3 only, I assign the partition 3 aggregator owner to partition 3 worker)?

I hope in your comprehension and I hope I somehow caught your attention, even if for a brief moment. Ask me if something is not clear ;)

Cheers!

~~~~~~~~~~~~~~~~~~~

Ing. Alessio Arleo

Dottorando in Ingegneria Industriale e dell’Informazione

Dottore Magistrale in Ingegneria Informatica e dell’Automazione
Dottore in Ingegneria Informatica ed Elettronica

Linkedin: it.linkedin.com/in/IngArleo <http://it.linkedin.com/in/IngArleo>
Skype: Ing. Alessio Arleo

Tel: +39 075 5853920
Cell: +39 349 0575782

~~~~~~~~~~~~~~~~~~~