You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Arjun Sharma <as...@gmail.com> on 2015/03/25 19:51:32 UTC
Custom assignment of partitions to workers
Hi,
I understand we can override the GraphPartitionerFactory class in order to
achieve custom partitioning of vertices over partitions. Is there a way to
do the same to enable assigning partitions to workers in a custom way
(e.g., partition n should be assigned to worker m)? The reason is that it
is more beneficial to have partitions that are close to each other in terms
of the graph structure placed close to each other on the same worker to
minimize network traffic. Otherwise, benefits of
overriding GraphPartitionerFactory may be lost. Let us assume that there
is an external preprocessing step that outputs the desired assignment, so
can we materialize that assignment over Giraph workers?
Thanks!
Re: Custom assignment of partitions to workers
Posted by Lukas Nalezenec <lu...@firma.seznam.cz>.
Its same.
GraphPartitionerFactory has got two methods. One for
WorkerGraphPartitioner and second for MasterGraphPartitioner.
public interface GraphPartitionerFactory ... {
MasterGraphPartitioner<I, V, E> createMasterGraphPartitioner();
WorkerGraphPartitioner<I, V, E> createWorkerGraphPartitioner();
}
Lukas
On 25.3.2015 20:57, Arjun Sharma wrote:
> Thanks for your reply! I understand we can supply the
> GraphPartitionerFactory class to Giraph using the
> giraph.graphPartitionerFactoryClass class. Is there a way to supply
> the MasterGraphPartitioner class?
>
> Thanks!
>
> On Wed, Mar 25, 2015 at 12:54 PM, Lukas Nalezenec
> <lukas.nalezenec@firma.seznam.cz
> <ma...@firma.seznam.cz>> wrote:
> >
> > Hi,
> >
> > There are two interfaces:
> > WorkerGraphPartitioner - Maps vertexes to partitions
> > MasterGraphPartitioner - Maps partitions to workers.
> > So you need custom MasterGraphPartitioner.
> > You dont need any external preprocessing step.
> >
> > Lukas
> >
> >
> > On 25.3.2015 19:51, Arjun Sharma wrote:
> >
> > Hi,
> >
> > I understand we can override the GraphPartitionerFactory class in
> order to achieve custom partitioning of vertices over partitions. Is
> there a way to do the same to enable assigning partitions to workers
> in a custom way (e.g., partition n should be assigned to worker m)?
> The reason is that it is more beneficial to have partitions that are
> close to each other in terms of the graph structure placed close to
> each other on the same worker to minimize network traffic. Otherwise,
> benefits of overriding GraphPartitionerFactory may be lost. Let us
> assume that there is an external preprocessing step that outputs the
> desired assignment, so can we materialize that assignment over Giraph
> workers?
> >
> > Thanks!
> >
> >
Re: Custom assignment of partitions to workers
Posted by Arjun Sharma <as...@gmail.com>.
Thanks for your reply! I understand we can supply the
GraphPartitionerFactory class to Giraph using the
giraph.graphPartitionerFactoryClass class. Is there a way to supply the
MasterGraphPartitioner class?
Thanks!
On Wed, Mar 25, 2015 at 12:54 PM, Lukas Nalezenec <
lukas.nalezenec@firma.seznam.cz> wrote:
>
> Hi,
>
> There are two interfaces:
> WorkerGraphPartitioner - Maps vertexes to partitions
> MasterGraphPartitioner - Maps partitions to workers.
> So you need custom MasterGraphPartitioner.
> You dont need any external preprocessing step.
>
> Lukas
>
>
> On 25.3.2015 19:51, Arjun Sharma wrote:
>
> Hi,
>
> I understand we can override the GraphPartitionerFactory class in order
to achieve custom partitioning of vertices over partitions. Is there a way
to do the same to enable assigning partitions to workers in a custom way
(e.g., partition n should be assigned to worker m)? The reason is that it
is more beneficial to have partitions that are close to each other in terms
of the graph structure placed close to each other on the same worker to
minimize network traffic. Otherwise, benefits of overriding
GraphPartitionerFactory may be lost. Let us assume that there is an
external preprocessing step that outputs the desired assignment, so can we
materialize that assignment over Giraph workers?
>
> Thanks!
>
>
Re: Custom assignment of partitions to workers
Posted by Lukas Nalezenec <lu...@firma.seznam.cz>.
Hi,
There are two interfaces:
WorkerGraphPartitioner - Maps vertexes to partitions
MasterGraphPartitioner - Maps partitions to workers.
So you need custom MasterGraphPartitioner.
You dont need any external preprocessing step.
Lukas
On 25.3.2015 19:51, Arjun Sharma wrote:
> Hi,
>
> I understand we can override the GraphPartitionerFactory class in
> order to achieve custom partitioning of vertices over partitions. Is
> there a way to do the same to enable assigning partitions to workers
> in a custom way (e.g., partition n should be assigned to worker m)?
> The reason is that it is more beneficial to have partitions that are
> close to each other in terms of the graph structure placed close to
> each other on the same worker to minimize network traffic. Otherwise,
> benefits of overriding GraphPartitionerFactory may be lost. Let us
> assume that there is an external preprocessing step that outputs the
> desired assignment, so can we materialize that assignment over Giraph
> workers?
>
> Thanks!