You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Marco Aurelio Barbosa Fagnani Lotz <m....@stu12.qmul.ac.uk> on 2013/08/17 21:20:37 UTC
Workers input splits and MasterCompute communication
Hello all :)
In what class the workers actually get the input file splits from the file system?
Is it possible to a MasterCompute class object to have access/communication with the workers in that job? I though about using aggregators, but then I assumed that aggregators actually work with vertices compute() (and related methods) and not with the worker itself.
When I mean workers I don't mean the vertices in each worker, but the object that runs the compute for all the vertices in that worker.
Best Regards,
Marco Lotz
Re: Workers input splits and MasterCompute communication
Posted by Avery Ching <ac...@apache.org>.
That makes sense, since the Context doesn't have a real InputSplit (it's
a Giraph one - see BspInputSplit).
What information are you trying to get out of the input splits? Giraph
workers can process an arbitrary number of input splits (0 or more), so
I don't think this will be useful.
You can use Configuration if you need to set some information at runtime.
Avery
On 8/19/13 9:14 AM, Marco Aurelio Barbosa Fagnani Lotz wrote:
> Hello all :)
>
> I am having problems calling getContext().getInputSplit(); inside the
> compute() method in the workers.
>
> It always returns as if it didn't get any split at all, since
> inputSplit.getLocations() returns without the hosts that should have
> that split as local and inputSplit.getLength() returns 0.
>
> Should there be any initialization to the Workers context so that I
> can get this information?
> Is there anyway to access the jobContext from the workers or the Master?
>
> Best Regards,
> Marco Lotz
>
> ------------------------------------------------------------------------
> *From:* Marco Aurelio Barbosa Fagnani Lotz <m....@stu12.qmul.ac.uk>
> *Sent:* 17 August 2013 20:20
> *To:* user@giraph.apache.org
> *Subject:* Workers input splits and MasterCompute communication
> Hello all :)
>
> In what class the workers actually get the input file splits from the
> file system?
>
> Is it possible to a MasterCompute class object to have
> access/communication with the workers in that job? I though about
> using aggregators, but then I assumed that aggregators actually work
> with vertices compute() (and related methods) and not with the worker
> itself.
>
> When I mean workers I don't mean the vertices in each worker, but the
> object that runs the compute for all the vertices in that worker.
>
> Best Regards,
> Marco Lotz
RE: Workers input splits and MasterCompute communication
Posted by Marco Aurelio Barbosa Fagnani Lotz <m....@stu12.qmul.ac.uk>.
Hello all :)
I am having problems calling getContext().getInputSplit(); inside the compute() method in the workers.
It always returns as if it didn't get any split at all, since inputSplit.getLocations() returns without the hosts that should have that split as local and inputSplit.getLength() returns 0.
Should there be any initialization to the Workers context so that I can get this information?
Is there anyway to access the jobContext from the workers or the Master?
Best Regards,
Marco Lotz
________________________________
From: Marco Aurelio Barbosa Fagnani Lotz <m....@stu12.qmul.ac.uk>
Sent: 17 August 2013 20:20
To: user@giraph.apache.org
Subject: Workers input splits and MasterCompute communication
Hello all :)
In what class the workers actually get the input file splits from the file system?
Is it possible to a MasterCompute class object to have access/communication with the workers in that job? I though about using aggregators, but then I assumed that aggregators actually work with vertices compute() (and related methods) and not with the worker itself.
When I mean workers I don't mean the vertices in each worker, but the object that runs the compute for all the vertices in that worker.
Best Regards,
Marco Lotz