You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by jamborta <ja...@gmail.com> on 2009/12/15 19:10:02 UTC
RandomUtils.useTestSeed() - Taste libraries
hi there,
sorry, one more question today.
I have this evaluator which based on your
AbstractDifferenceRecommenderEvaluator, I wanted to run it using
RandomUtils.useTestSeed(), to make sure that I get the same results. for
example if I have this method:
public void recommend() {
try {
final DataModel model = new FileDataModel(new
File("./data/test_data.data"));
RecommenderBuilder build = new SVDBuilder();
DataModelBuilder model2 = null;
IREvaluatorNoParallel evaluate = new IREvaluatorNoParallel();
evaluate.evaluate(build, model2, model, 0.6, 1.0);
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (TasteException e) {
e.printStackTrace();
}
}
I would like to have the same result if I call it more than one times.
however, if i do this:
RandomUtils.useTestSeed();
test.recommend();
test.recommend();
It gives me a differrent result for the second times. but if I run it again,
it gives me the same different results. not sure if i understand how i
works. is it possible to do this?
thanks
--
View this message in context: http://old.nabble.com/RandomUtils.useTestSeed%28%29---Taste-libraries-tp26799137p26799137.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: RandomUtils.useTestSeed() - Taste libraries
Posted by jamborta <ja...@gmail.com>.
I guess all I need is to run the same method while changing the parameter.
srowen wrote:
>
> No, do you really need to do this though?
> It's possible to add a method to do this but I'm curious about the use
> case, whether we can come up with another way to solve your issue.
>
> On Tue, Dec 15, 2009 at 6:23 PM, jamborta <ja...@gmail.com> wrote:
>>
>> thanks. so it's not possible to reset the seeds? or start with new ones?
>
>
--
View this message in context: http://old.nabble.com/RandomUtils.useTestSeed%28%29---Taste-libraries-tp26799137p26800154.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: RandomUtils.useTestSeed() - Taste libraries
Posted by Sean Owen <sr...@gmail.com>.
I think the issue is different here, and the proposed change as I
understand it doesn't result in the desired behavior anyway. Even if
you could seed the RNG. That said I am not really against changes
here.
On Tue, Dec 15, 2009 at 8:16 PM, Ted Dunning <te...@gmail.com> wrote:
> Sean,
>
> Remember my grumpiness about not having full injection of the random number
> generator?
>
> That would help here.
Re: RandomUtils.useTestSeed() - Taste libraries
Posted by Ted Dunning <te...@gmail.com>.
Sean,
Remember my grumpiness about not having full injection of the random number
generator?
That would help here.
On Tue, Dec 15, 2009 at 11:19 AM, Sean Owen <sr...@gmail.com> wrote:
> It's hard to guarantee since, if you switch in another implementation
> that also uses the random number generator, then it will still change
> the samples that are taken.
>
> If you're concerned about this I would suggest running tests over more
> data, or running more iterations, in order to obtain a more reliable
> result. The sampling shouldn't make a significant difference.
>
> On Tue, Dec 15, 2009 at 7:15 PM, jamborta <ja...@gmail.com> wrote:
> >
> > in this case i try to evaluate an algorithm and tune some parameters, but
> if
> > I don't know that I use the same
> > trainging/test set, it's hard to compare the results. I guess I could
> keep
> > track of how many times I called
> > the method, but that would make it complicated to administer.
> >
> >
> >
> > srowen wrote:
> >>
> >> No, do you really need to do this though?
> >> It's possible to add a method to do this but I'm curious about the use
> >> case, whether we can come up with another way to solve your issue.
> >>
> >> On Tue, Dec 15, 2009 at 6:23 PM, jamborta <ja...@gmail.com> wrote:
> >>>
> >>> thanks. so it's not possible to reset the seeds? or start with new
> ones?
> >>
> >>
> >
> > --
> > View this message in context:
> http://old.nabble.com/RandomUtils.useTestSeed%28%29---Taste-libraries-tp26799137p26800133.html
> > Sent from the Mahout User List mailing list archive at Nabble.com.
> >
> >
>
--
Ted Dunning, CTO
DeepDyve
Re: RandomUtils.useTestSeed() - Taste libraries
Posted by jamborta <ja...@gmail.com>.
ok. thanks very much.
srowen wrote:
>
> It's hard to guarantee since, if you switch in another implementation
> that also uses the random number generator, then it will still change
> the samples that are taken.
>
> If you're concerned about this I would suggest running tests over more
> data, or running more iterations, in order to obtain a more reliable
> result. The sampling shouldn't make a significant difference.
>
> On Tue, Dec 15, 2009 at 7:15 PM, jamborta <ja...@gmail.com> wrote:
>>
>> in this case i try to evaluate an algorithm and tune some parameters, but
>> if
>> I don't know that I use the same
>> trainging/test set, it's hard to compare the results. I guess I could
>> keep
>> track of how many times I called
>> the method, but that would make it complicated to administer.
>>
>>
>>
>> srowen wrote:
>>>
>>> No, do you really need to do this though?
>>> It's possible to add a method to do this but I'm curious about the use
>>> case, whether we can come up with another way to solve your issue.
>>>
>>> On Tue, Dec 15, 2009 at 6:23 PM, jamborta <ja...@gmail.com> wrote:
>>>>
>>>> thanks. so it's not possible to reset the seeds? or start with new
>>>> ones?
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/RandomUtils.useTestSeed%28%29---Taste-libraries-tp26799137p26800133.html
>> Sent from the Mahout User List mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://old.nabble.com/RandomUtils.useTestSeed%28%29---Taste-libraries-tp26799137p26800243.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: RandomUtils.useTestSeed() - Taste libraries
Posted by Sean Owen <sr...@gmail.com>.
It's hard to guarantee since, if you switch in another implementation
that also uses the random number generator, then it will still change
the samples that are taken.
If you're concerned about this I would suggest running tests over more
data, or running more iterations, in order to obtain a more reliable
result. The sampling shouldn't make a significant difference.
On Tue, Dec 15, 2009 at 7:15 PM, jamborta <ja...@gmail.com> wrote:
>
> in this case i try to evaluate an algorithm and tune some parameters, but if
> I don't know that I use the same
> trainging/test set, it's hard to compare the results. I guess I could keep
> track of how many times I called
> the method, but that would make it complicated to administer.
>
>
>
> srowen wrote:
>>
>> No, do you really need to do this though?
>> It's possible to add a method to do this but I'm curious about the use
>> case, whether we can come up with another way to solve your issue.
>>
>> On Tue, Dec 15, 2009 at 6:23 PM, jamborta <ja...@gmail.com> wrote:
>>>
>>> thanks. so it's not possible to reset the seeds? or start with new ones?
>>
>>
>
> --
> View this message in context: http://old.nabble.com/RandomUtils.useTestSeed%28%29---Taste-libraries-tp26799137p26800133.html
> Sent from the Mahout User List mailing list archive at Nabble.com.
>
>
Re: RandomUtils.useTestSeed() - Taste libraries
Posted by jamborta <ja...@gmail.com>.
in this case i try to evaluate an algorithm and tune some parameters, but if
I don't know that I use the same
trainging/test set, it's hard to compare the results. I guess I could keep
track of how many times I called
the method, but that would make it complicated to administer.
srowen wrote:
>
> No, do you really need to do this though?
> It's possible to add a method to do this but I'm curious about the use
> case, whether we can come up with another way to solve your issue.
>
> On Tue, Dec 15, 2009 at 6:23 PM, jamborta <ja...@gmail.com> wrote:
>>
>> thanks. so it's not possible to reset the seeds? or start with new ones?
>
>
--
View this message in context: http://old.nabble.com/RandomUtils.useTestSeed%28%29---Taste-libraries-tp26799137p26800133.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: RandomUtils.useTestSeed() - Taste libraries
Posted by Sean Owen <sr...@gmail.com>.
No, do you really need to do this though?
It's possible to add a method to do this but I'm curious about the use
case, whether we can come up with another way to solve your issue.
On Tue, Dec 15, 2009 at 6:23 PM, jamborta <ja...@gmail.com> wrote:
>
> thanks. so it's not possible to reset the seeds? or start with new ones?
Re: RandomUtils.useTestSeed() - Taste libraries
Posted by jamborta <ja...@gmail.com>.
thanks. so it's not possible to reset the seeds? or start with new ones?
srowen wrote:
>
> Yes, the evaluator uses randomness to pick test data and so on. You
> will get different results when you run the evaluation multiple times.
> However yes, as you see, if you fix the random number generator seed,
> then you should at least still see the same first result, and same
> second result, and so on. It does not somehow reset the seeds between
> evaluation runs, no.
>
> On Tue, Dec 15, 2009 at 6:10 PM, jamborta <ja...@gmail.com> wrote:
>>
>> hi there,
>>
>> sorry, one more question today.
>>
>> I have this evaluator which based on your
>> AbstractDifferenceRecommenderEvaluator, I wanted to run it using
>> RandomUtils.useTestSeed(), to make sure that I get the same results. for
>> example if I have this method:
>> public void recommend() {
>> try {
>> final DataModel model = new FileDataModel(new
>> File("./data/test_data.data"));
>> RecommenderBuilder build = new SVDBuilder();
>> DataModelBuilder model2 = null;
>> IREvaluatorNoParallel evaluate = new IREvaluatorNoParallel();
>> evaluate.evaluate(build, model2, model, 0.6, 1.0);
>> } catch (FileNotFoundException e) {
>> e.printStackTrace();
>>
>> } catch (TasteException e) {
>> e.printStackTrace();
>> }
>> }
>> I would like to have the same result if I call it more than one times.
>> however, if i do this:
>>
>> RandomUtils.useTestSeed();
>> test.recommend();
>> test.recommend();
>>
>> It gives me a differrent result for the second times. but if I run it
>> again,
>> it gives me the same different results. not sure if i understand how i
>> works. is it possible to do this?
>>
>> thanks
>> --
>> View this message in context:
>> http://old.nabble.com/RandomUtils.useTestSeed%28%29---Taste-libraries-tp26799137p26799137.html
>> Sent from the Mahout User List mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://old.nabble.com/RandomUtils.useTestSeed%28%29---Taste-libraries-tp26799137p26799342.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: RandomUtils.useTestSeed() - Taste libraries
Posted by Sean Owen <sr...@gmail.com>.
Yes, the evaluator uses randomness to pick test data and so on. You
will get different results when you run the evaluation multiple times.
However yes, as you see, if you fix the random number generator seed,
then you should at least still see the same first result, and same
second result, and so on. It does not somehow reset the seeds between
evaluation runs, no.
On Tue, Dec 15, 2009 at 6:10 PM, jamborta <ja...@gmail.com> wrote:
>
> hi there,
>
> sorry, one more question today.
>
> I have this evaluator which based on your
> AbstractDifferenceRecommenderEvaluator, I wanted to run it using
> RandomUtils.useTestSeed(), to make sure that I get the same results. for
> example if I have this method:
> public void recommend() {
> try {
> final DataModel model = new FileDataModel(new
> File("./data/test_data.data"));
> RecommenderBuilder build = new SVDBuilder();
> DataModelBuilder model2 = null;
> IREvaluatorNoParallel evaluate = new IREvaluatorNoParallel();
> evaluate.evaluate(build, model2, model, 0.6, 1.0);
> } catch (FileNotFoundException e) {
> e.printStackTrace();
>
> } catch (TasteException e) {
> e.printStackTrace();
> }
> }
> I would like to have the same result if I call it more than one times.
> however, if i do this:
>
> RandomUtils.useTestSeed();
> test.recommend();
> test.recommend();
>
> It gives me a differrent result for the second times. but if I run it again,
> it gives me the same different results. not sure if i understand how i
> works. is it possible to do this?
>
> thanks
> --
> View this message in context: http://old.nabble.com/RandomUtils.useTestSeed%28%29---Taste-libraries-tp26799137p26799137.html
> Sent from the Mahout User List mailing list archive at Nabble.com.
>
>