You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samoa.apache.org by Gianmarco De Francisci Morales <gd...@apache.org> on 2016/04/03 09:40:39 UTC
Re: Running SAMOA with Storm
Hi Jean Paul,
I cannot reproduce the issue on my setup, the default bagging works fine on
my cluster.
Have you validated that your Storm cluster works for other applications?
-- Gianmarco
On Thu, Mar 24, 2016 at 1:40 PM, Nicolas Kourtellis <nk...@gmail.com>
wrote:
> Hi Jean,
>
> The example you mentioned running is using yahoo.labs code.
> Did you find these on the samoa - apache page? Can you point to the
> location?
>
> We may need to update those with the apache versions.
>
> In any case, this generator should now be on the master of apache repo.
>
> Thanks,
>
> Nicolas
>
> On Tue, Mar 8, 2016 at 2:54 AM, Jean Paul Barddal <jp...@gmail.com>
> wrote:
>
> > Nope.
> > I’ve tested the default configuration provided in examples in the samoa
> > page, such as -s
> > (com.yahoo.labs.samoa.moa.streams.generators.RandomTreeGenerator -c 2 -o
> 10
> > -u 10).
> >
> > Cheers,
> >
> > Jean
> >
> > > On Mar 8, 2016, at 12:18 AM, Gianmarco De Francisci Morales <
> > gdfm@apache.org> wrote:
> > >
> > > The Boosting is not very well tested, so I would expect some bugs
> there.
> > > Bagging is a bit more well tested.
> > > Are you using a dataset with a very large number of attributes?
> > >
> > > -- Gianmarco
> > >
> > > On Mon, Mar 7, 2016 at 2:55 AM, Jean Paul Barddal <jpbarddal@gmail.com
> >
> > > wrote:
> > >
> > >> Hello Gianmarco,
> > >>
> > >> I’ve tried both Bagging and Boosting.
> > >> They end up in a GC error pretty quickly.
> > >>
> > >> Jean
> > >>
> > >>
> > >>> On Mar 7, 2016, at 1:03 AM, Gianmarco De Francisci Morales <
> > >> gdfm@apache.org> wrote:
> > >>>
> > >>> Hi Jean Paul,
> > >>>
> > >>> Which ensemble are you running? Bagging?
> > >>> And what's the ensemble size?
> > >>>
> > >>> Nicolas, if I remember correctly we do not have acking for Storm
> > enabled
> > >> in
> > >>> the current version on master.
> > >>> Maybe this is why not all instances are executed?
> > >>> If so, I could port the acking we have on the VHT branch on master.
> > >>>
> > >>> Cheers,
> > >>>
> > >>> -- Gianmarco
> > >>>
> > >>> On Thu, Feb 25, 2016 at 4:23 PM, Jean Paul Barddal <
> > jpbarddal@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Hello again everyone,
> > >>>>
> > >>>> I managed to improve a little my situation.
> > >>>> Now, the problem only occurs with ensembles.
> > >>>> I am sending too screenshots of two issues I currently have.
> > >>>> If the amount of instances is to big (1 million or more), then I ran
> > >> into
> > >>>> a GC problem.
> > >>>> If less, sometimes the evaluation node does not execute all the
> > >> instances.
> > >>>> I’ve left the topology running over night and it was still stuck.
> > >>>>
> > >>>> http://i67.tinypic.com/10cqnvp.png
> > >>>> http://i66.tinypic.com/f9mwrc.png
> > >>>>
> > >>>> Cheers,
> > >>>>
> > >>>> Jean
> > >>>>
> > >>>>
> > >>>>> On Feb 23, 2016, at 8:53 AM, Jean Paul Barddal <
> jpbarddal@gmail.com>
> > >>>> wrote:
> > >>>>>
> > >>>>> Dear Nicolas and Gianmarco,
> > >>>>>
> > >>>>> Thanks for the replies!
> > >>>>>
> > >>>>> The screenshot can be found at: http://i68.tinypic.com/25jdu9v.png
> > >>>>>
> > >>>>> Yes, I have tried a few examples from the guide. I’ve tried VHT and
> > >>>> Bagging over a RandomTreeGenerator.
> > >>>>>
> > >>>>> Sometimes it works, but then I get a GC (garbage collector error)
> > too.
> > >>>>>
> > >>>>> Gianmarco, I’m running Storm 0.9.5 and SAMOA 0.0.4.
> > >>>>>
> > >>>>> Cheers,
> > >>>>>
> > >>>>> Jean
> > >>>>>
> > >>>>>> On Feb 22, 2016, at 6:11 PM, Gianmarco De Francisci Morales <
> > >>>> gdfm@apache.org> wrote:
> > >>>>>>
> > >>>>>> Also, which version of Storm are you using?
> > >>>>>>
> > >>>>>> -- Gianmarco
> > >>>>>>
> > >>>>>> On Mon, Feb 22, 2016 at 1:01 PM, Nicolas Kourtellis <
> > >>>> nkourtellis@gmail.com>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Hi Jean,
> > >>>>>>> Thanks for your interest in SAMOA!
> > >>>>>>> I dont think the attachment came through.
> > >>>>>>> Maybe you can post it in an online storage and send a link?
> > >>>>>>>
> > >>>>>>> Regarding your question, have you tried the basic examples from
> the
> > >>>> guide?
> > >>>>>>> E.g., the VHT in storm mode?
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>>
> > >>>>>>> Nicolas
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Mon, Feb 22, 2016 at 5:21 PM, Jean Paul Barddal <
> > >>>> jpbarddal@gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Hello guys,
> > >>>>>>>>
> > >>>>>>>> I am trying to set up a SAMOA environment and I'm having some
> > >> issues.
> > >>>>>>>> I have the storm up and running and managed to download, compile
> > and
> > >>>> run
> > >>>>>>>> SAMOA in local configuration.
> > >>>>>>>> Now I'm trying to run it in cluster mode.
> > >>>>>>>> As you can see from the attached screnshot, the topology is up
> and
> > >>>>>>>> running, but the spouts do not send any data to the bolts and so
> > on.
> > >>>>>>>>
> > >>>>>>>> Ideas?
> > >>>>>>>>
> > >>>>>>>> Cheers,
> > >>>>>>>>
> > >>>>>>>> Jean
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>> Nicolas Kourtellis
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
>
>
> --
> Nicolas Kourtellis
>
Re: Running SAMOA with Storm
Posted by Gianmarco De Francisci Morales <gd...@apache.org>.
I'm running the master version of SAMOA from github and Storm 0.9.6.
In my case they are VMs, 4GB of RAM per core.
I think Nicolas has a different setup.
No changes on the Storm configuration.
Cheers,
-- Gianmarco
On Sun, Apr 3, 2016 at 10:47 AM, Jean Paul Barddal <jp...@gmail.com>
wrote:
> Hi Gianmarco and Nicolas,
>
> Yes, it does.
> Can you tell me which version of SAMOA and Storm you have running in your
> cluster?
> I will try to set up a new cluster from scratch to see if I manage to get
> it to work.
> Also, can you tell me if your clusters’ nodes are VMs? And you many RAM
> each has?
> Finally, do you have any changes to your garbage collection on storm, or
> everything is as default?
>
>
> Nicolas, I honestly don’t recall where I saw this example using com.yahoo
> classes.
> Should I use others instead?
>
> Cheers,
>
> Jean
>
>
> > On Apr 3, 2016, at 7:40 PM, Gianmarco De Francisci Morales <
> gdfm@apache.org> wrote:
> >
> > Hi Jean Paul,
> > I cannot reproduce the issue on my setup, the default bagging works fine
> on
> > my cluster.
> > Have you validated that your Storm cluster works for other applications?
> >
> > -- Gianmarco
> >
> > On Thu, Mar 24, 2016 at 1:40 PM, Nicolas Kourtellis <
> nkourtellis@gmail.com>
> > wrote:
> >
> >> Hi Jean,
> >>
> >> The example you mentioned running is using yahoo.labs code.
> >> Did you find these on the samoa - apache page? Can you point to the
> >> location?
> >>
> >> We may need to update those with the apache versions.
> >>
> >> In any case, this generator should now be on the master of apache repo.
> >>
> >> Thanks,
> >>
> >> Nicolas
> >>
> >> On Tue, Mar 8, 2016 at 2:54 AM, Jean Paul Barddal <jp...@gmail.com>
> >> wrote:
> >>
> >>> Nope.
> >>> I’ve tested the default configuration provided in examples in the samoa
> >>> page, such as -s
> >>> (com.yahoo.labs.samoa.moa.streams.generators.RandomTreeGenerator -c 2
> -o
> >> 10
> >>> -u 10).
> >>>
> >>> Cheers,
> >>>
> >>> Jean
> >>>
> >>>> On Mar 8, 2016, at 12:18 AM, Gianmarco De Francisci Morales <
> >>> gdfm@apache.org> wrote:
> >>>>
> >>>> The Boosting is not very well tested, so I would expect some bugs
> >> there.
> >>>> Bagging is a bit more well tested.
> >>>> Are you using a dataset with a very large number of attributes?
> >>>>
> >>>> -- Gianmarco
> >>>>
> >>>> On Mon, Mar 7, 2016 at 2:55 AM, Jean Paul Barddal <
> jpbarddal@gmail.com
> >>>
> >>>> wrote:
> >>>>
> >>>>> Hello Gianmarco,
> >>>>>
> >>>>> I’ve tried both Bagging and Boosting.
> >>>>> They end up in a GC error pretty quickly.
> >>>>>
> >>>>> Jean
> >>>>>
> >>>>>
> >>>>>> On Mar 7, 2016, at 1:03 AM, Gianmarco De Francisci Morales <
> >>>>> gdfm@apache.org> wrote:
> >>>>>>
> >>>>>> Hi Jean Paul,
> >>>>>>
> >>>>>> Which ensemble are you running? Bagging?
> >>>>>> And what's the ensemble size?
> >>>>>>
> >>>>>> Nicolas, if I remember correctly we do not have acking for Storm
> >>> enabled
> >>>>> in
> >>>>>> the current version on master.
> >>>>>> Maybe this is why not all instances are executed?
> >>>>>> If so, I could port the acking we have on the VHT branch on master.
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> -- Gianmarco
> >>>>>>
> >>>>>> On Thu, Feb 25, 2016 at 4:23 PM, Jean Paul Barddal <
> >>> jpbarddal@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hello again everyone,
> >>>>>>>
> >>>>>>> I managed to improve a little my situation.
> >>>>>>> Now, the problem only occurs with ensembles.
> >>>>>>> I am sending too screenshots of two issues I currently have.
> >>>>>>> If the amount of instances is to big (1 million or more), then I
> ran
> >>>>> into
> >>>>>>> a GC problem.
> >>>>>>> If less, sometimes the evaluation node does not execute all the
> >>>>> instances.
> >>>>>>> I’ve left the topology running over night and it was still stuck.
> >>>>>>>
> >>>>>>> http://i67.tinypic.com/10cqnvp.png
> >>>>>>> http://i66.tinypic.com/f9mwrc.png
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>>
> >>>>>>> Jean
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Feb 23, 2016, at 8:53 AM, Jean Paul Barddal <
> >> jpbarddal@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Dear Nicolas and Gianmarco,
> >>>>>>>>
> >>>>>>>> Thanks for the replies!
> >>>>>>>>
> >>>>>>>> The screenshot can be found at:
> http://i68.tinypic.com/25jdu9v.png
> >>>>>>>>
> >>>>>>>> Yes, I have tried a few examples from the guide. I’ve tried VHT
> and
> >>>>>>> Bagging over a RandomTreeGenerator.
> >>>>>>>>
> >>>>>>>> Sometimes it works, but then I get a GC (garbage collector error)
> >>> too.
> >>>>>>>>
> >>>>>>>> Gianmarco, I’m running Storm 0.9.5 and SAMOA 0.0.4.
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>>
> >>>>>>>> Jean
> >>>>>>>>
> >>>>>>>>> On Feb 22, 2016, at 6:11 PM, Gianmarco De Francisci Morales <
> >>>>>>> gdfm@apache.org> wrote:
> >>>>>>>>>
> >>>>>>>>> Also, which version of Storm are you using?
> >>>>>>>>>
> >>>>>>>>> -- Gianmarco
> >>>>>>>>>
> >>>>>>>>> On Mon, Feb 22, 2016 at 1:01 PM, Nicolas Kourtellis <
> >>>>>>> nkourtellis@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Jean,
> >>>>>>>>>> Thanks for your interest in SAMOA!
> >>>>>>>>>> I dont think the attachment came through.
> >>>>>>>>>> Maybe you can post it in an online storage and send a link?
> >>>>>>>>>>
> >>>>>>>>>> Regarding your question, have you tried the basic examples from
> >> the
> >>>>>>> guide?
> >>>>>>>>>> E.g., the VHT in storm mode?
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>>
> >>>>>>>>>> Nicolas
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Feb 22, 2016 at 5:21 PM, Jean Paul Barddal <
> >>>>>>> jpbarddal@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hello guys,
> >>>>>>>>>>>
> >>>>>>>>>>> I am trying to set up a SAMOA environment and I'm having some
> >>>>> issues.
> >>>>>>>>>>> I have the storm up and running and managed to download,
> compile
> >>> and
> >>>>>>> run
> >>>>>>>>>>> SAMOA in local configuration.
> >>>>>>>>>>> Now I'm trying to run it in cluster mode.
> >>>>>>>>>>> As you can see from the attached screnshot, the topology is up
> >> and
> >>>>>>>>>>> running, but the spouts do not send any data to the bolts and
> so
> >>> on.
> >>>>>>>>>>>
> >>>>>>>>>>> Ideas?
> >>>>>>>>>>>
> >>>>>>>>>>> Cheers,
> >>>>>>>>>>>
> >>>>>>>>>>> Jean
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Nicolas Kourtellis
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
> >>
> >> --
> >> Nicolas Kourtellis
> >>
>
>
Re: Running SAMOA with Storm
Posted by Jean Paul Barddal <jp...@gmail.com>.
Dear Nicolas and Gianmarco,
Thanks for the feedback and for the script!
I’ll try and will get back to you :)
Regards,
Jean
> On Apr 14, 2016, at 9:59 PM, Nicolas Kourtellis <nk...@gmail.com> wrote:
>
> <samoa-storm-steps.txt>
Re: Running SAMOA with Storm
Posted by Nicolas Kourtellis <nk...@gmail.com>.
Hi Jean Paul,
I am also using the current version of the master on the incubator (i.e.,
0.4.0) and Storm 0.9.3.
If you want to confirm the setup, check the attached for some steps I wrote
a while ago on how to
get Samoa configured to play with a Storm configuration:
(note that I didn't change much of the default settings)
My server has physical cpus so I am not running it in VMs, with a total of
65GB.
Regarding the yahoo examples, there are probably applicable but refer to
older versions of Samoa.
So I would stick to the current ones from the Apache Incubator page of
Samoa.
Hope this helps!
Nicolas
On Sun, Apr 3, 2016 at 9:51 AM, Gianmarco De Francisci Morales <gdfm@gdfm.me
> wrote:
> I'm running the master version of SAMOA from github and Storm 0.9.6.
>
> In my case they are VMs, 4GB of RAM per core.
> I think Nicolas has a different setup.
> No changes on the Storm configuration.
>
> -- Gianmarco
>
> On Sun, Apr 3, 2016 at 10:47 AM, Jean Paul Barddal <jp...@gmail.com>
> wrote:
>
> > Hi Gianmarco and Nicolas,
> >
> > Yes, it does.
> > Can you tell me which version of SAMOA and Storm you have running in your
> > cluster?
> > I will try to set up a new cluster from scratch to see if I manage to get
> > it to work.
> > Also, can you tell me if your clusters’ nodes are VMs? And you many RAM
> > each has?
> > Finally, do you have any changes to your garbage collection on storm, or
> > everything is as default?
> >
> >
> > Nicolas, I honestly don’t recall where I saw this example using com.yahoo
> > classes.
> > Should I use others instead?
> >
> > Cheers,
> >
> > Jean
> >
> >
> > > On Apr 3, 2016, at 7:40 PM, Gianmarco De Francisci Morales <
> > gdfm@apache.org> wrote:
> > >
> > > Hi Jean Paul,
> > > I cannot reproduce the issue on my setup, the default bagging works
> fine
> > on
> > > my cluster.
> > > Have you validated that your Storm cluster works for other
> applications?
> > >
> > > -- Gianmarco
> > >
> > > On Thu, Mar 24, 2016 at 1:40 PM, Nicolas Kourtellis <
> > nkourtellis@gmail.com>
> > > wrote:
> > >
> > >> Hi Jean,
> > >>
> > >> The example you mentioned running is using yahoo.labs code.
> > >> Did you find these on the samoa - apache page? Can you point to the
> > >> location?
> > >>
> > >> We may need to update those with the apache versions.
> > >>
> > >> In any case, this generator should now be on the master of apache
> repo.
> > >>
> > >> Thanks,
> > >>
> > >> Nicolas
> > >>
> > >> On Tue, Mar 8, 2016 at 2:54 AM, Jean Paul Barddal <
> jpbarddal@gmail.com>
> > >> wrote:
> > >>
> > >>> Nope.
> > >>> I’ve tested the default configuration provided in examples in the
> samoa
> > >>> page, such as -s
> > >>> (com.yahoo.labs.samoa.moa.streams.generators.RandomTreeGenerator -c 2
> > -o
> > >> 10
> > >>> -u 10).
> > >>>
> > >>> Cheers,
> > >>>
> > >>> Jean
> > >>>
> > >>>> On Mar 8, 2016, at 12:18 AM, Gianmarco De Francisci Morales <
> > >>> gdfm@apache.org> wrote:
> > >>>>
> > >>>> The Boosting is not very well tested, so I would expect some bugs
> > >> there.
> > >>>> Bagging is a bit more well tested.
> > >>>> Are you using a dataset with a very large number of attributes?
> > >>>>
> > >>>> -- Gianmarco
> > >>>>
> > >>>> On Mon, Mar 7, 2016 at 2:55 AM, Jean Paul Barddal <
> > jpbarddal@gmail.com
> > >>>
> > >>>> wrote:
> > >>>>
> > >>>>> Hello Gianmarco,
> > >>>>>
> > >>>>> I’ve tried both Bagging and Boosting.
> > >>>>> They end up in a GC error pretty quickly.
> > >>>>>
> > >>>>> Jean
> > >>>>>
> > >>>>>
> > >>>>>> On Mar 7, 2016, at 1:03 AM, Gianmarco De Francisci Morales <
> > >>>>> gdfm@apache.org> wrote:
> > >>>>>>
> > >>>>>> Hi Jean Paul,
> > >>>>>>
> > >>>>>> Which ensemble are you running? Bagging?
> > >>>>>> And what's the ensemble size?
> > >>>>>>
> > >>>>>> Nicolas, if I remember correctly we do not have acking for Storm
> > >>> enabled
> > >>>>> in
> > >>>>>> the current version on master.
> > >>>>>> Maybe this is why not all instances are executed?
> > >>>>>> If so, I could port the acking we have on the VHT branch on
> master.
> > >>>>>>
> > >>>>>> Cheers,
> > >>>>>>
> > >>>>>> -- Gianmarco
> > >>>>>>
> > >>>>>> On Thu, Feb 25, 2016 at 4:23 PM, Jean Paul Barddal <
> > >>> jpbarddal@gmail.com>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Hello again everyone,
> > >>>>>>>
> > >>>>>>> I managed to improve a little my situation.
> > >>>>>>> Now, the problem only occurs with ensembles.
> > >>>>>>> I am sending too screenshots of two issues I currently have.
> > >>>>>>> If the amount of instances is to big (1 million or more), then I
> > ran
> > >>>>> into
> > >>>>>>> a GC problem.
> > >>>>>>> If less, sometimes the evaluation node does not execute all the
> > >>>>> instances.
> > >>>>>>> I’ve left the topology running over night and it was still stuck.
> > >>>>>>>
> > >>>>>>> http://i67.tinypic.com/10cqnvp.png
> > >>>>>>> http://i66.tinypic.com/f9mwrc.png
> > >>>>>>>
> > >>>>>>> Cheers,
> > >>>>>>>
> > >>>>>>> Jean
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> On Feb 23, 2016, at 8:53 AM, Jean Paul Barddal <
> > >> jpbarddal@gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> Dear Nicolas and Gianmarco,
> > >>>>>>>>
> > >>>>>>>> Thanks for the replies!
> > >>>>>>>>
> > >>>>>>>> The screenshot can be found at:
> > http://i68.tinypic.com/25jdu9v.png
> > >>>>>>>>
> > >>>>>>>> Yes, I have tried a few examples from the guide. I’ve tried VHT
> > and
> > >>>>>>> Bagging over a RandomTreeGenerator.
> > >>>>>>>>
> > >>>>>>>> Sometimes it works, but then I get a GC (garbage collector
> error)
> > >>> too.
> > >>>>>>>>
> > >>>>>>>> Gianmarco, I’m running Storm 0.9.5 and SAMOA 0.0.4.
> > >>>>>>>>
> > >>>>>>>> Cheers,
> > >>>>>>>>
> > >>>>>>>> Jean
> > >>>>>>>>
> > >>>>>>>>> On Feb 22, 2016, at 6:11 PM, Gianmarco De Francisci Morales <
> > >>>>>>> gdfm@apache.org> wrote:
> > >>>>>>>>>
> > >>>>>>>>> Also, which version of Storm are you using?
> > >>>>>>>>>
> > >>>>>>>>> -- Gianmarco
> > >>>>>>>>>
> > >>>>>>>>> On Mon, Feb 22, 2016 at 1:01 PM, Nicolas Kourtellis <
> > >>>>>>> nkourtellis@gmail.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi Jean,
> > >>>>>>>>>> Thanks for your interest in SAMOA!
> > >>>>>>>>>> I dont think the attachment came through.
> > >>>>>>>>>> Maybe you can post it in an online storage and send a link?
> > >>>>>>>>>>
> > >>>>>>>>>> Regarding your question, have you tried the basic examples
> from
> > >> the
> > >>>>>>> guide?
> > >>>>>>>>>> E.g., the VHT in storm mode?
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks,
> > >>>>>>>>>>
> > >>>>>>>>>> Nicolas
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Mon, Feb 22, 2016 at 5:21 PM, Jean Paul Barddal <
> > >>>>>>> jpbarddal@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hello guys,
> > >>>>>>>>>>>
> > >>>>>>>>>>> I am trying to set up a SAMOA environment and I'm having some
> > >>>>> issues.
> > >>>>>>>>>>> I have the storm up and running and managed to download,
> > compile
> > >>> and
> > >>>>>>> run
> > >>>>>>>>>>> SAMOA in local configuration.
> > >>>>>>>>>>> Now I'm trying to run it in cluster mode.
> > >>>>>>>>>>> As you can see from the attached screnshot, the topology is
> up
> > >> and
> > >>>>>>>>>>> running, but the spouts do not send any data to the bolts and
> > so
> > >>> on.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Ideas?
> > >>>>>>>>>>>
> > >>>>>>>>>>> Cheers,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Jean
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> --
> > >>>>>>>>>> Nicolas Kourtellis
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>>
> > >>>
> > >>>
> > >>
> > >>
> > >> --
> > >> Nicolas Kourtellis
> > >>
> >
> >
>
--
Nicolas Kourtellis
Re: Running SAMOA with Storm
Posted by Gianmarco De Francisci Morales <gd...@gdfm.me>.
I'm running the master version of SAMOA from github and Storm 0.9.6.
In my case they are VMs, 4GB of RAM per core.
I think Nicolas has a different setup.
No changes on the Storm configuration.
-- Gianmarco
On Sun, Apr 3, 2016 at 10:47 AM, Jean Paul Barddal <jp...@gmail.com>
wrote:
> Hi Gianmarco and Nicolas,
>
> Yes, it does.
> Can you tell me which version of SAMOA and Storm you have running in your
> cluster?
> I will try to set up a new cluster from scratch to see if I manage to get
> it to work.
> Also, can you tell me if your clusters’ nodes are VMs? And you many RAM
> each has?
> Finally, do you have any changes to your garbage collection on storm, or
> everything is as default?
>
>
> Nicolas, I honestly don’t recall where I saw this example using com.yahoo
> classes.
> Should I use others instead?
>
> Cheers,
>
> Jean
>
>
> > On Apr 3, 2016, at 7:40 PM, Gianmarco De Francisci Morales <
> gdfm@apache.org> wrote:
> >
> > Hi Jean Paul,
> > I cannot reproduce the issue on my setup, the default bagging works fine
> on
> > my cluster.
> > Have you validated that your Storm cluster works for other applications?
> >
> > -- Gianmarco
> >
> > On Thu, Mar 24, 2016 at 1:40 PM, Nicolas Kourtellis <
> nkourtellis@gmail.com>
> > wrote:
> >
> >> Hi Jean,
> >>
> >> The example you mentioned running is using yahoo.labs code.
> >> Did you find these on the samoa - apache page? Can you point to the
> >> location?
> >>
> >> We may need to update those with the apache versions.
> >>
> >> In any case, this generator should now be on the master of apache repo.
> >>
> >> Thanks,
> >>
> >> Nicolas
> >>
> >> On Tue, Mar 8, 2016 at 2:54 AM, Jean Paul Barddal <jp...@gmail.com>
> >> wrote:
> >>
> >>> Nope.
> >>> I’ve tested the default configuration provided in examples in the samoa
> >>> page, such as -s
> >>> (com.yahoo.labs.samoa.moa.streams.generators.RandomTreeGenerator -c 2
> -o
> >> 10
> >>> -u 10).
> >>>
> >>> Cheers,
> >>>
> >>> Jean
> >>>
> >>>> On Mar 8, 2016, at 12:18 AM, Gianmarco De Francisci Morales <
> >>> gdfm@apache.org> wrote:
> >>>>
> >>>> The Boosting is not very well tested, so I would expect some bugs
> >> there.
> >>>> Bagging is a bit more well tested.
> >>>> Are you using a dataset with a very large number of attributes?
> >>>>
> >>>> -- Gianmarco
> >>>>
> >>>> On Mon, Mar 7, 2016 at 2:55 AM, Jean Paul Barddal <
> jpbarddal@gmail.com
> >>>
> >>>> wrote:
> >>>>
> >>>>> Hello Gianmarco,
> >>>>>
> >>>>> I’ve tried both Bagging and Boosting.
> >>>>> They end up in a GC error pretty quickly.
> >>>>>
> >>>>> Jean
> >>>>>
> >>>>>
> >>>>>> On Mar 7, 2016, at 1:03 AM, Gianmarco De Francisci Morales <
> >>>>> gdfm@apache.org> wrote:
> >>>>>>
> >>>>>> Hi Jean Paul,
> >>>>>>
> >>>>>> Which ensemble are you running? Bagging?
> >>>>>> And what's the ensemble size?
> >>>>>>
> >>>>>> Nicolas, if I remember correctly we do not have acking for Storm
> >>> enabled
> >>>>> in
> >>>>>> the current version on master.
> >>>>>> Maybe this is why not all instances are executed?
> >>>>>> If so, I could port the acking we have on the VHT branch on master.
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> -- Gianmarco
> >>>>>>
> >>>>>> On Thu, Feb 25, 2016 at 4:23 PM, Jean Paul Barddal <
> >>> jpbarddal@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hello again everyone,
> >>>>>>>
> >>>>>>> I managed to improve a little my situation.
> >>>>>>> Now, the problem only occurs with ensembles.
> >>>>>>> I am sending too screenshots of two issues I currently have.
> >>>>>>> If the amount of instances is to big (1 million or more), then I
> ran
> >>>>> into
> >>>>>>> a GC problem.
> >>>>>>> If less, sometimes the evaluation node does not execute all the
> >>>>> instances.
> >>>>>>> I’ve left the topology running over night and it was still stuck.
> >>>>>>>
> >>>>>>> http://i67.tinypic.com/10cqnvp.png
> >>>>>>> http://i66.tinypic.com/f9mwrc.png
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>>
> >>>>>>> Jean
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Feb 23, 2016, at 8:53 AM, Jean Paul Barddal <
> >> jpbarddal@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Dear Nicolas and Gianmarco,
> >>>>>>>>
> >>>>>>>> Thanks for the replies!
> >>>>>>>>
> >>>>>>>> The screenshot can be found at:
> http://i68.tinypic.com/25jdu9v.png
> >>>>>>>>
> >>>>>>>> Yes, I have tried a few examples from the guide. I’ve tried VHT
> and
> >>>>>>> Bagging over a RandomTreeGenerator.
> >>>>>>>>
> >>>>>>>> Sometimes it works, but then I get a GC (garbage collector error)
> >>> too.
> >>>>>>>>
> >>>>>>>> Gianmarco, I’m running Storm 0.9.5 and SAMOA 0.0.4.
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>>
> >>>>>>>> Jean
> >>>>>>>>
> >>>>>>>>> On Feb 22, 2016, at 6:11 PM, Gianmarco De Francisci Morales <
> >>>>>>> gdfm@apache.org> wrote:
> >>>>>>>>>
> >>>>>>>>> Also, which version of Storm are you using?
> >>>>>>>>>
> >>>>>>>>> -- Gianmarco
> >>>>>>>>>
> >>>>>>>>> On Mon, Feb 22, 2016 at 1:01 PM, Nicolas Kourtellis <
> >>>>>>> nkourtellis@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Jean,
> >>>>>>>>>> Thanks for your interest in SAMOA!
> >>>>>>>>>> I dont think the attachment came through.
> >>>>>>>>>> Maybe you can post it in an online storage and send a link?
> >>>>>>>>>>
> >>>>>>>>>> Regarding your question, have you tried the basic examples from
> >> the
> >>>>>>> guide?
> >>>>>>>>>> E.g., the VHT in storm mode?
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>>
> >>>>>>>>>> Nicolas
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Feb 22, 2016 at 5:21 PM, Jean Paul Barddal <
> >>>>>>> jpbarddal@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hello guys,
> >>>>>>>>>>>
> >>>>>>>>>>> I am trying to set up a SAMOA environment and I'm having some
> >>>>> issues.
> >>>>>>>>>>> I have the storm up and running and managed to download,
> compile
> >>> and
> >>>>>>> run
> >>>>>>>>>>> SAMOA in local configuration.
> >>>>>>>>>>> Now I'm trying to run it in cluster mode.
> >>>>>>>>>>> As you can see from the attached screnshot, the topology is up
> >> and
> >>>>>>>>>>> running, but the spouts do not send any data to the bolts and
> so
> >>> on.
> >>>>>>>>>>>
> >>>>>>>>>>> Ideas?
> >>>>>>>>>>>
> >>>>>>>>>>> Cheers,
> >>>>>>>>>>>
> >>>>>>>>>>> Jean
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Nicolas Kourtellis
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
> >>
> >> --
> >> Nicolas Kourtellis
> >>
>
>
Re: Running SAMOA with Storm
Posted by Jean Paul Barddal <jp...@gmail.com>.
Hi Gianmarco and Nicolas,
Yes, it does.
Can you tell me which version of SAMOA and Storm you have running in your cluster?
I will try to set up a new cluster from scratch to see if I manage to get it to work.
Also, can you tell me if your clusters’ nodes are VMs? And you many RAM each has?
Finally, do you have any changes to your garbage collection on storm, or everything is as default?
Nicolas, I honestly don’t recall where I saw this example using com.yahoo classes.
Should I use others instead?
Cheers,
Jean
> On Apr 3, 2016, at 7:40 PM, Gianmarco De Francisci Morales <gd...@apache.org> wrote:
>
> Hi Jean Paul,
> I cannot reproduce the issue on my setup, the default bagging works fine on
> my cluster.
> Have you validated that your Storm cluster works for other applications?
>
> -- Gianmarco
>
> On Thu, Mar 24, 2016 at 1:40 PM, Nicolas Kourtellis <nk...@gmail.com>
> wrote:
>
>> Hi Jean,
>>
>> The example you mentioned running is using yahoo.labs code.
>> Did you find these on the samoa - apache page? Can you point to the
>> location?
>>
>> We may need to update those with the apache versions.
>>
>> In any case, this generator should now be on the master of apache repo.
>>
>> Thanks,
>>
>> Nicolas
>>
>> On Tue, Mar 8, 2016 at 2:54 AM, Jean Paul Barddal <jp...@gmail.com>
>> wrote:
>>
>>> Nope.
>>> I’ve tested the default configuration provided in examples in the samoa
>>> page, such as -s
>>> (com.yahoo.labs.samoa.moa.streams.generators.RandomTreeGenerator -c 2 -o
>> 10
>>> -u 10).
>>>
>>> Cheers,
>>>
>>> Jean
>>>
>>>> On Mar 8, 2016, at 12:18 AM, Gianmarco De Francisci Morales <
>>> gdfm@apache.org> wrote:
>>>>
>>>> The Boosting is not very well tested, so I would expect some bugs
>> there.
>>>> Bagging is a bit more well tested.
>>>> Are you using a dataset with a very large number of attributes?
>>>>
>>>> -- Gianmarco
>>>>
>>>> On Mon, Mar 7, 2016 at 2:55 AM, Jean Paul Barddal <jpbarddal@gmail.com
>>>
>>>> wrote:
>>>>
>>>>> Hello Gianmarco,
>>>>>
>>>>> I’ve tried both Bagging and Boosting.
>>>>> They end up in a GC error pretty quickly.
>>>>>
>>>>> Jean
>>>>>
>>>>>
>>>>>> On Mar 7, 2016, at 1:03 AM, Gianmarco De Francisci Morales <
>>>>> gdfm@apache.org> wrote:
>>>>>>
>>>>>> Hi Jean Paul,
>>>>>>
>>>>>> Which ensemble are you running? Bagging?
>>>>>> And what's the ensemble size?
>>>>>>
>>>>>> Nicolas, if I remember correctly we do not have acking for Storm
>>> enabled
>>>>> in
>>>>>> the current version on master.
>>>>>> Maybe this is why not all instances are executed?
>>>>>> If so, I could port the acking we have on the VHT branch on master.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> -- Gianmarco
>>>>>>
>>>>>> On Thu, Feb 25, 2016 at 4:23 PM, Jean Paul Barddal <
>>> jpbarddal@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello again everyone,
>>>>>>>
>>>>>>> I managed to improve a little my situation.
>>>>>>> Now, the problem only occurs with ensembles.
>>>>>>> I am sending too screenshots of two issues I currently have.
>>>>>>> If the amount of instances is to big (1 million or more), then I ran
>>>>> into
>>>>>>> a GC problem.
>>>>>>> If less, sometimes the evaluation node does not execute all the
>>>>> instances.
>>>>>>> I’ve left the topology running over night and it was still stuck.
>>>>>>>
>>>>>>> http://i67.tinypic.com/10cqnvp.png
>>>>>>> http://i66.tinypic.com/f9mwrc.png
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Jean
>>>>>>>
>>>>>>>
>>>>>>>> On Feb 23, 2016, at 8:53 AM, Jean Paul Barddal <
>> jpbarddal@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Dear Nicolas and Gianmarco,
>>>>>>>>
>>>>>>>> Thanks for the replies!
>>>>>>>>
>>>>>>>> The screenshot can be found at: http://i68.tinypic.com/25jdu9v.png
>>>>>>>>
>>>>>>>> Yes, I have tried a few examples from the guide. I’ve tried VHT and
>>>>>>> Bagging over a RandomTreeGenerator.
>>>>>>>>
>>>>>>>> Sometimes it works, but then I get a GC (garbage collector error)
>>> too.
>>>>>>>>
>>>>>>>> Gianmarco, I’m running Storm 0.9.5 and SAMOA 0.0.4.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Jean
>>>>>>>>
>>>>>>>>> On Feb 22, 2016, at 6:11 PM, Gianmarco De Francisci Morales <
>>>>>>> gdfm@apache.org> wrote:
>>>>>>>>>
>>>>>>>>> Also, which version of Storm are you using?
>>>>>>>>>
>>>>>>>>> -- Gianmarco
>>>>>>>>>
>>>>>>>>> On Mon, Feb 22, 2016 at 1:01 PM, Nicolas Kourtellis <
>>>>>>> nkourtellis@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Jean,
>>>>>>>>>> Thanks for your interest in SAMOA!
>>>>>>>>>> I dont think the attachment came through.
>>>>>>>>>> Maybe you can post it in an online storage and send a link?
>>>>>>>>>>
>>>>>>>>>> Regarding your question, have you tried the basic examples from
>> the
>>>>>>> guide?
>>>>>>>>>> E.g., the VHT in storm mode?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Nicolas
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Feb 22, 2016 at 5:21 PM, Jean Paul Barddal <
>>>>>>> jpbarddal@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello guys,
>>>>>>>>>>>
>>>>>>>>>>> I am trying to set up a SAMOA environment and I'm having some
>>>>> issues.
>>>>>>>>>>> I have the storm up and running and managed to download, compile
>>> and
>>>>>>> run
>>>>>>>>>>> SAMOA in local configuration.
>>>>>>>>>>> Now I'm trying to run it in cluster mode.
>>>>>>>>>>> As you can see from the attached screnshot, the topology is up
>> and
>>>>>>>>>>> running, but the spouts do not send any data to the bolts and so
>>> on.
>>>>>>>>>>>
>>>>>>>>>>> Ideas?
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>>
>>>>>>>>>>> Jean
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Nicolas Kourtellis
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>>
>>
>> --
>> Nicolas Kourtellis
>>