You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Matthew Cornell <ma...@matthewcornell.org> on 2014/08/26 16:36:58 UTC
How do I validate customArguments?
Hi again. My application needs to pass in a String argument to the
computation which each Vertex needs access to. (The argument is a list of
the form "[item1, item2, ...]".) I found --customArguments (which I set in
my tests via conf.set(<arg_name>, <arg_val>)) but I need to check that it's
properly formatted. Where do I do that? The only thing I thought of is to
specify a DefaultMasterCompute subclass whose initialize() does the check,
but all the initialize() examples do is register aggregators; none of them
check args or do anything else. Thanks in advance! -- matt
--
Matthew Cornell | matt@matthewcornell.org | 413-626-3621 | 34 Dickinson
Street, Amherst MA 01002 | matthewcornell.org
Re: How do I validate customArguments?
Posted by Matthew Saltz <sa...@gmail.com>.
No worries. Just by the way, I realized after I sent that that using the
"public static int numPreprocessingSteps" to store the value in the
MasterCompute class doesn't work; you need to register a permanent
aggregator to hold on to it, if you need to.
Best,
Matthew
On Wed, Sep 10, 2014 at 3:15 PM, Matthew Cornell <ma...@matthewcornell.org>
wrote:
> Sorry for the long delay, Matthew. That's really helpful. Right now I'm
> stuck on apparently running out of memory on our little cluster, but the
> log messages are confusing. I'm putting together a question, but in the
> meantime I'll try one of the simpler examples such as degree count to see
> if /anything/ will run against my graph, which is very small (100K and
> edges nodes). -- matt
>
> On Thu, Aug 28, 2014 at 2:26 PM, Matthew Saltz <sa...@gmail.com> wrote:
>
>> Matt,
>>
>> I'm not sure if you've resolved this problem already or not, but if you
>> haven't: The initialize() method isn't limited to registering aggregators,
>> and in fact, in my project I use it to do exactly what you're describing to
>> check and load custom configuration parameters. Inside the initialize()
>> method, I do this:
>>
>> * String numPreprocessingStepsConf =
>> getConf().get(NUMBER_OF_PREPROCESSING_STEPS_CONF_OPT);*
>> * numPreprocessingSteps = (numPreprocessingStepsConf != null) ?*
>> * Integer.parseInt(numPreprocessingStepsConf.trim()) :*
>> * DEFAULT_NUMBER_OF_PREPROCESSING_STEPS;*
>> * System.out.println("Number of preprocessing steps: " +
>> numPreprocessingSteps);*
>>
>> where at the class level I declare:
>>
>> public static final String NUMBER_OF_PREPROCESSING_STEPS_CONF_OPT =
>> "wcc.numPreprocessingSteps";
>> public static final int DEFAULT_NUMBER_OF_PREPROCESSING_STEPS = 1;
>> public static int numPreprocessingSteps;
>>
>> To set the property, I use the option "-ca
>> wcc.numPreprocessingSteps=<number of steps I want>". If you need to check
>> that it's properly formatted and not store them, this is a fine place to do
>> it as well, given that it's run before the input superstep (see the giraph
>> code in BspServiceMaster, line 1617 in the stable 1.1.0 release). What
>> happens is that on the master, the MasterThread calls coordinateSuperstep()
>> on a BspServiceMaster object, which checks if it's the input superstep, and
>> if so, calls initialize() on the MasterCompute object (created in the
>> becomeMaster() method of BspServiceMaster).
>>
>> Hope this helps,
>> Matthew
>>
>>
>>
>> On Tue, Aug 26, 2014 at 4:36 PM, Matthew Cornell <matt@matthewcornell.org
>> > wrote:
>>
>>> Hi again. My application needs to pass in a String argument to the
>>> computation which each Vertex needs access to. (The argument is a list of
>>> the form "[item1, item2, ...]".) I found --customArguments (which I set in
>>> my tests via conf.set(<arg_name>, <arg_val>)) but I need to check that it's
>>> properly formatted. Where do I do that? The only thing I thought of is to
>>> specify a DefaultMasterCompute subclass whose initialize() does the check,
>>> but all the initialize() examples do is register aggregators; none of them
>>> check args or do anything else. Thanks in advance! -- matt
>>>
>>> --
>>> Matthew Cornell | matt@matthewcornell.org | 413-626-3621 | 34 Dickinson
>>> Street, Amherst MA 01002 | matthewcornell.org
>>>
>>
>>
>
>
> --
> Matthew Cornell | matt@matthewcornell.org | 413-626-3621 | 34 Dickinson
> Street, Amherst MA 01002 | matthewcornell.org
>
Re: How do I validate customArguments?
Posted by Matthew Cornell <ma...@matthewcornell.org>.
Sorry for the long delay, Matthew. That's really helpful. Right now I'm
stuck on apparently running out of memory on our little cluster, but the
log messages are confusing. I'm putting together a question, but in the
meantime I'll try one of the simpler examples such as degree count to see
if /anything/ will run against my graph, which is very small (100K and
edges nodes). -- matt
On Thu, Aug 28, 2014 at 2:26 PM, Matthew Saltz <sa...@gmail.com> wrote:
> Matt,
>
> I'm not sure if you've resolved this problem already or not, but if you
> haven't: The initialize() method isn't limited to registering aggregators,
> and in fact, in my project I use it to do exactly what you're describing to
> check and load custom configuration parameters. Inside the initialize()
> method, I do this:
>
> * String numPreprocessingStepsConf =
> getConf().get(NUMBER_OF_PREPROCESSING_STEPS_CONF_OPT);*
> * numPreprocessingSteps = (numPreprocessingStepsConf != null) ?*
> * Integer.parseInt(numPreprocessingStepsConf.trim()) :*
> * DEFAULT_NUMBER_OF_PREPROCESSING_STEPS;*
> * System.out.println("Number of preprocessing steps: " +
> numPreprocessingSteps);*
>
> where at the class level I declare:
>
> public static final String NUMBER_OF_PREPROCESSING_STEPS_CONF_OPT =
> "wcc.numPreprocessingSteps";
> public static final int DEFAULT_NUMBER_OF_PREPROCESSING_STEPS = 1;
> public static int numPreprocessingSteps;
>
> To set the property, I use the option "-ca
> wcc.numPreprocessingSteps=<number of steps I want>". If you need to check
> that it's properly formatted and not store them, this is a fine place to do
> it as well, given that it's run before the input superstep (see the giraph
> code in BspServiceMaster, line 1617 in the stable 1.1.0 release). What
> happens is that on the master, the MasterThread calls coordinateSuperstep()
> on a BspServiceMaster object, which checks if it's the input superstep, and
> if so, calls initialize() on the MasterCompute object (created in the
> becomeMaster() method of BspServiceMaster).
>
> Hope this helps,
> Matthew
>
>
>
> On Tue, Aug 26, 2014 at 4:36 PM, Matthew Cornell <ma...@matthewcornell.org>
> wrote:
>
>> Hi again. My application needs to pass in a String argument to the
>> computation which each Vertex needs access to. (The argument is a list of
>> the form "[item1, item2, ...]".) I found --customArguments (which I set in
>> my tests via conf.set(<arg_name>, <arg_val>)) but I need to check that it's
>> properly formatted. Where do I do that? The only thing I thought of is to
>> specify a DefaultMasterCompute subclass whose initialize() does the check,
>> but all the initialize() examples do is register aggregators; none of them
>> check args or do anything else. Thanks in advance! -- matt
>>
>> --
>> Matthew Cornell | matt@matthewcornell.org | 413-626-3621 | 34 Dickinson
>> Street, Amherst MA 01002 | matthewcornell.org
>>
>
>
--
Matthew Cornell | matt@matthewcornell.org | 413-626-3621 | 34 Dickinson
Street, Amherst MA 01002 | matthewcornell.org
Re: How do I validate customArguments?
Posted by Matthew Saltz <sa...@gmail.com>.
Matt,
I'm not sure if you've resolved this problem already or not, but if you
haven't: The initialize() method isn't limited to registering aggregators,
and in fact, in my project I use it to do exactly what you're describing to
check and load custom configuration parameters. Inside the initialize()
method, I do this:
* String numPreprocessingStepsConf =
getConf().get(NUMBER_OF_PREPROCESSING_STEPS_CONF_OPT);*
* numPreprocessingSteps = (numPreprocessingStepsConf != null) ?*
* Integer.parseInt(numPreprocessingStepsConf.trim()) :*
* DEFAULT_NUMBER_OF_PREPROCESSING_STEPS;*
* System.out.println("Number of preprocessing steps: " +
numPreprocessingSteps);*
where at the class level I declare:
public static final String NUMBER_OF_PREPROCESSING_STEPS_CONF_OPT =
"wcc.numPreprocessingSteps";
public static final int DEFAULT_NUMBER_OF_PREPROCESSING_STEPS = 1;
public static int numPreprocessingSteps;
To set the property, I use the option "-ca
wcc.numPreprocessingSteps=<number of steps I want>". If you need to check
that it's properly formatted and not store them, this is a fine place to do
it as well, given that it's run before the input superstep (see the giraph
code in BspServiceMaster, line 1617 in the stable 1.1.0 release). What
happens is that on the master, the MasterThread calls coordinateSuperstep()
on a BspServiceMaster object, which checks if it's the input superstep, and
if so, calls initialize() on the MasterCompute object (created in the
becomeMaster() method of BspServiceMaster).
Hope this helps,
Matthew
On Tue, Aug 26, 2014 at 4:36 PM, Matthew Cornell <ma...@matthewcornell.org>
wrote:
> Hi again. My application needs to pass in a String argument to the
> computation which each Vertex needs access to. (The argument is a list of
> the form "[item1, item2, ...]".) I found --customArguments (which I set in
> my tests via conf.set(<arg_name>, <arg_val>)) but I need to check that it's
> properly formatted. Where do I do that? The only thing I thought of is to
> specify a DefaultMasterCompute subclass whose initialize() does the check,
> but all the initialize() examples do is register aggregators; none of them
> check args or do anything else. Thanks in advance! -- matt
>
> --
> Matthew Cornell | matt@matthewcornell.org | 413-626-3621 | 34 Dickinson
> Street, Amherst MA 01002 | matthewcornell.org
>