You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@whirr.apache.org by John Conwell <jo...@iamjohn.me> on 2011/06/10 19:37:32 UTC
Is Service.launchCluster thread safe?
In 0.4.0, is Service.launchCluster(ClusterSpec) thread safe? Through the
API I'm spinning up 3 clusters, one after another, and I'd like to change
the code to launch them in parallel, but wanted to check if this is
threadsafe.
--
Thanks,
John C
Re: Is Service.launchCluster thread safe?
Posted by Andrei Savu <sa...@gmail.com>.
Side note: if you are starting multiple Hadoop clusters at the same time you
should know that you can start only one proxy at a time (they are using the
same hardcoded port).
Thanks to this you have to interact with the started cluster in a serial
fashion. This is a known issue.
On Jun 15, 2011 8:34 PM, "John Conwell" <jo...@iamjohn.me> wrote:
> Yea, I can do that. What section would you want it in? Maybe a new section
> on using whirr via its API?
>
> On Tue, Jun 14, 2011 at 5:20 PM, Tom White <to...@gmail.com> wrote:
>
>> On Tue, Jun 14, 2011 at 3:46 PM, John Conwell <jo...@iamjohn.me> wrote:
>> > So as an FYI, I just tested using the whirr API to start multiple
>> clusters
>> > at the same time using Futures, and it (seems to) works great. really
>> cuts
>> > down on the time to ramp up a set of clusters (like 4 or more). Yay
>>
>> Great. This would be a nice thing to put on the wiki or in the docs as
>> a usage example. Would you like to do this?
>>
>> Thanks,
>> Tom
>>
>> >
>> > On Fri, Jun 10, 2011 at 11:38 AM, Andrei Savu <sa...@gmail.com>
>> wrote:
>> >>
>> >> John,
>> >>
>> >> I don't think we've checked this. Could you open an issue? We should
at
>> >> least update the docs.
>> >>
>> >> To be safe you should create multiple instances, one for each thread.
I
>> >> don't think you need to worry about the amount of memory used.
>> >>
>> >> In 0.5.0 we've done some performance improvements that are relevant to
>> >> you.
>> >>
>> >> Sent from my phone.
>> >>
>> >> Cheers,
>> >>
>> >> -- Andrei
>> >>
>> >> On Jun 10, 2011 8:38 PM, "John Conwell" <jo...@iamjohn.me> wrote:
>> >> > In 0.4.0, is Service.launchCluster(ClusterSpec) thread safe? Through
>> the
>> >> > API I'm spinning up 3 clusters, one after another, and I'd like to
>> >> > change
>> >> > the code to launch them in parallel, but wanted to check if this is
>> >> > threadsafe.
>> >> >
>> >> > --
>> >> >
>> >> > Thanks,
>> >> > John C
>> >
>> >
>> >
>> > --
>> >
>> > Thanks,
>> > John C
>> >
>>
>
>
>
> --
>
> Thanks,
> John C
Re: Is Service.launchCluster thread safe?
Posted by Tom White <to...@gmail.com>.
Either's OK.
On Wed, Jun 15, 2011 at 11:08 AM, John Conwell <jo...@iamjohn.me> wrote:
> So you want wiki changes made via documentation in svn, not on the wiki
> itself? Looks like I need to upgrade my solr whirr code to 0.5.0
>
> On Wed, Jun 15, 2011 at 10:41 AM, Tom White <to...@gmail.com> wrote:
>>
>> There's a page at src/site/xdoc/api-guide.xml, which might be a good
>> place for this.
>>
>> Thanks!
>> Tom
>>
>> On Wed, Jun 15, 2011 at 10:34 AM, John Conwell <jo...@iamjohn.me> wrote:
>> > Yea, I can do that. What section would you want it in? Maybe a new
>> > section
>> > on using whirr via its API?
>> >
>> > On Tue, Jun 14, 2011 at 5:20 PM, Tom White <to...@gmail.com>
>> > wrote:
>> >>
>> >> On Tue, Jun 14, 2011 at 3:46 PM, John Conwell <jo...@iamjohn.me> wrote:
>> >> > So as an FYI, I just tested using the whirr API to start multiple
>> >> > clusters
>> >> > at the same time using Futures, and it (seems to) works great.
>> >> > really
>> >> > cuts
>> >> > down on the time to ramp up a set of clusters (like 4 or more). Yay
>> >>
>> >> Great. This would be a nice thing to put on the wiki or in the docs as
>> >> a usage example. Would you like to do this?
>> >>
>> >> Thanks,
>> >> Tom
>> >>
>> >> >
>> >> > On Fri, Jun 10, 2011 at 11:38 AM, Andrei Savu <sa...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> John,
>> >> >>
>> >> >> I don't think we've checked this. Could you open an issue? We should
>> >> >> at
>> >> >> least update the docs.
>> >> >>
>> >> >> To be safe you should create multiple instances, one for each
>> >> >> thread. I
>> >> >> don't think you need to worry about the amount of memory used.
>> >> >>
>> >> >> In 0.5.0 we've done some performance improvements that are relevant
>> >> >> to
>> >> >> you.
>> >> >>
>> >> >> Sent from my phone.
>> >> >>
>> >> >> Cheers,
>> >> >>
>> >> >> -- Andrei
>> >> >>
>> >> >> On Jun 10, 2011 8:38 PM, "John Conwell" <jo...@iamjohn.me> wrote:
>> >> >> > In 0.4.0, is Service.launchCluster(ClusterSpec) thread safe?
>> >> >> > Through
>> >> >> > the
>> >> >> > API I'm spinning up 3 clusters, one after another, and I'd like to
>> >> >> > change
>> >> >> > the code to launch them in parallel, but wanted to check if this
>> >> >> > is
>> >> >> > threadsafe.
>> >> >> >
>> >> >> > --
>> >> >> >
>> >> >> > Thanks,
>> >> >> > John C
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> >
>> >> > Thanks,
>> >> > John C
>> >> >
>> >
>> >
>> >
>> > --
>> >
>> > Thanks,
>> > John C
>> >
>
>
>
> --
>
> Thanks,
> John C
>
Re: Is Service.launchCluster thread safe?
Posted by John Conwell <jo...@iamjohn.me>.
So you want wiki changes made via documentation in svn, not on the wiki
itself? Looks like I need to upgrade my solr whirr code to 0.5.0
On Wed, Jun 15, 2011 at 10:41 AM, Tom White <to...@gmail.com> wrote:
> There's a page at src/site/xdoc/api-guide.xml, which might be a good
> place for this.
>
> Thanks!
> Tom
>
> On Wed, Jun 15, 2011 at 10:34 AM, John Conwell <jo...@iamjohn.me> wrote:
> > Yea, I can do that. What section would you want it in? Maybe a new
> section
> > on using whirr via its API?
> >
> > On Tue, Jun 14, 2011 at 5:20 PM, Tom White <to...@gmail.com>
> wrote:
> >>
> >> On Tue, Jun 14, 2011 at 3:46 PM, John Conwell <jo...@iamjohn.me> wrote:
> >> > So as an FYI, I just tested using the whirr API to start multiple
> >> > clusters
> >> > at the same time using Futures, and it (seems to) works great. really
> >> > cuts
> >> > down on the time to ramp up a set of clusters (like 4 or more). Yay
> >>
> >> Great. This would be a nice thing to put on the wiki or in the docs as
> >> a usage example. Would you like to do this?
> >>
> >> Thanks,
> >> Tom
> >>
> >> >
> >> > On Fri, Jun 10, 2011 at 11:38 AM, Andrei Savu <sa...@gmail.com>
> >> > wrote:
> >> >>
> >> >> John,
> >> >>
> >> >> I don't think we've checked this. Could you open an issue? We should
> at
> >> >> least update the docs.
> >> >>
> >> >> To be safe you should create multiple instances, one for each thread.
> I
> >> >> don't think you need to worry about the amount of memory used.
> >> >>
> >> >> In 0.5.0 we've done some performance improvements that are relevant
> to
> >> >> you.
> >> >>
> >> >> Sent from my phone.
> >> >>
> >> >> Cheers,
> >> >>
> >> >> -- Andrei
> >> >>
> >> >> On Jun 10, 2011 8:38 PM, "John Conwell" <jo...@iamjohn.me> wrote:
> >> >> > In 0.4.0, is Service.launchCluster(ClusterSpec) thread safe?
> Through
> >> >> > the
> >> >> > API I'm spinning up 3 clusters, one after another, and I'd like to
> >> >> > change
> >> >> > the code to launch them in parallel, but wanted to check if this is
> >> >> > threadsafe.
> >> >> >
> >> >> > --
> >> >> >
> >> >> > Thanks,
> >> >> > John C
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Thanks,
> >> > John C
> >> >
> >
> >
> >
> > --
> >
> > Thanks,
> > John C
> >
>
--
Thanks,
John C
Re: Is Service.launchCluster thread safe?
Posted by Tom White <to...@gmail.com>.
There's a page at src/site/xdoc/api-guide.xml, which might be a good
place for this.
Thanks!
Tom
On Wed, Jun 15, 2011 at 10:34 AM, John Conwell <jo...@iamjohn.me> wrote:
> Yea, I can do that. What section would you want it in? Maybe a new section
> on using whirr via its API?
>
> On Tue, Jun 14, 2011 at 5:20 PM, Tom White <to...@gmail.com> wrote:
>>
>> On Tue, Jun 14, 2011 at 3:46 PM, John Conwell <jo...@iamjohn.me> wrote:
>> > So as an FYI, I just tested using the whirr API to start multiple
>> > clusters
>> > at the same time using Futures, and it (seems to) works great. really
>> > cuts
>> > down on the time to ramp up a set of clusters (like 4 or more). Yay
>>
>> Great. This would be a nice thing to put on the wiki or in the docs as
>> a usage example. Would you like to do this?
>>
>> Thanks,
>> Tom
>>
>> >
>> > On Fri, Jun 10, 2011 at 11:38 AM, Andrei Savu <sa...@gmail.com>
>> > wrote:
>> >>
>> >> John,
>> >>
>> >> I don't think we've checked this. Could you open an issue? We should at
>> >> least update the docs.
>> >>
>> >> To be safe you should create multiple instances, one for each thread. I
>> >> don't think you need to worry about the amount of memory used.
>> >>
>> >> In 0.5.0 we've done some performance improvements that are relevant to
>> >> you.
>> >>
>> >> Sent from my phone.
>> >>
>> >> Cheers,
>> >>
>> >> -- Andrei
>> >>
>> >> On Jun 10, 2011 8:38 PM, "John Conwell" <jo...@iamjohn.me> wrote:
>> >> > In 0.4.0, is Service.launchCluster(ClusterSpec) thread safe? Through
>> >> > the
>> >> > API I'm spinning up 3 clusters, one after another, and I'd like to
>> >> > change
>> >> > the code to launch them in parallel, but wanted to check if this is
>> >> > threadsafe.
>> >> >
>> >> > --
>> >> >
>> >> > Thanks,
>> >> > John C
>> >
>> >
>> >
>> > --
>> >
>> > Thanks,
>> > John C
>> >
>
>
>
> --
>
> Thanks,
> John C
>
Re: Is Service.launchCluster thread safe?
Posted by John Conwell <jo...@iamjohn.me>.
Yea, I can do that. What section would you want it in? Maybe a new section
on using whirr via its API?
On Tue, Jun 14, 2011 at 5:20 PM, Tom White <to...@gmail.com> wrote:
> On Tue, Jun 14, 2011 at 3:46 PM, John Conwell <jo...@iamjohn.me> wrote:
> > So as an FYI, I just tested using the whirr API to start multiple
> clusters
> > at the same time using Futures, and it (seems to) works great. really
> cuts
> > down on the time to ramp up a set of clusters (like 4 or more). Yay
>
> Great. This would be a nice thing to put on the wiki or in the docs as
> a usage example. Would you like to do this?
>
> Thanks,
> Tom
>
> >
> > On Fri, Jun 10, 2011 at 11:38 AM, Andrei Savu <sa...@gmail.com>
> wrote:
> >>
> >> John,
> >>
> >> I don't think we've checked this. Could you open an issue? We should at
> >> least update the docs.
> >>
> >> To be safe you should create multiple instances, one for each thread. I
> >> don't think you need to worry about the amount of memory used.
> >>
> >> In 0.5.0 we've done some performance improvements that are relevant to
> >> you.
> >>
> >> Sent from my phone.
> >>
> >> Cheers,
> >>
> >> -- Andrei
> >>
> >> On Jun 10, 2011 8:38 PM, "John Conwell" <jo...@iamjohn.me> wrote:
> >> > In 0.4.0, is Service.launchCluster(ClusterSpec) thread safe? Through
> the
> >> > API I'm spinning up 3 clusters, one after another, and I'd like to
> >> > change
> >> > the code to launch them in parallel, but wanted to check if this is
> >> > threadsafe.
> >> >
> >> > --
> >> >
> >> > Thanks,
> >> > John C
> >
> >
> >
> > --
> >
> > Thanks,
> > John C
> >
>
--
Thanks,
John C
Re: Is Service.launchCluster thread safe?
Posted by Tom White <to...@gmail.com>.
On Tue, Jun 14, 2011 at 3:46 PM, John Conwell <jo...@iamjohn.me> wrote:
> So as an FYI, I just tested using the whirr API to start multiple clusters
> at the same time using Futures, and it (seems to) works great. really cuts
> down on the time to ramp up a set of clusters (like 4 or more). Yay
Great. This would be a nice thing to put on the wiki or in the docs as
a usage example. Would you like to do this?
Thanks,
Tom
>
> On Fri, Jun 10, 2011 at 11:38 AM, Andrei Savu <sa...@gmail.com> wrote:
>>
>> John,
>>
>> I don't think we've checked this. Could you open an issue? We should at
>> least update the docs.
>>
>> To be safe you should create multiple instances, one for each thread. I
>> don't think you need to worry about the amount of memory used.
>>
>> In 0.5.0 we've done some performance improvements that are relevant to
>> you.
>>
>> Sent from my phone.
>>
>> Cheers,
>>
>> -- Andrei
>>
>> On Jun 10, 2011 8:38 PM, "John Conwell" <jo...@iamjohn.me> wrote:
>> > In 0.4.0, is Service.launchCluster(ClusterSpec) thread safe? Through the
>> > API I'm spinning up 3 clusters, one after another, and I'd like to
>> > change
>> > the code to launch them in parallel, but wanted to check if this is
>> > threadsafe.
>> >
>> > --
>> >
>> > Thanks,
>> > John C
>
>
>
> --
>
> Thanks,
> John C
>
Re: Is Service.launchCluster thread safe?
Posted by John Conwell <jo...@iamjohn.me>.
So as an FYI, I just tested using the whirr API to start multiple clusters
at the same time using Futures, and it (seems to) works great. really cuts
down on the time to ramp up a set of clusters (like 4 or more). Yay
On Fri, Jun 10, 2011 at 11:38 AM, Andrei Savu <sa...@gmail.com> wrote:
> John,
>
> I don't think we've checked this. Could you open an issue? We should at
> least update the docs.
>
> To be safe you should create multiple instances, one for each thread. I
> don't think you need to worry about the amount of memory used.
>
> In 0.5.0 we've done some performance improvements that are relevant to you.
>
>
> Sent from my phone.
>
> Cheers,
>
> -- Andrei
> On Jun 10, 2011 8:38 PM, "John Conwell" <jo...@iamjohn.me> wrote:
> > In 0.4.0, is Service.launchCluster(ClusterSpec) thread safe? Through the
> > API I'm spinning up 3 clusters, one after another, and I'd like to change
> > the code to launch them in parallel, but wanted to check if this is
> > threadsafe.
> >
> > --
> >
> > Thanks,
> > John C
>
--
Thanks,
John C
Re: Is Service.launchCluster thread safe?
Posted by Andrei Savu <sa...@gmail.com>.
John,
I don't think we've checked this. Could you open an issue? We should at
least update the docs.
To be safe you should create multiple instances, one for each thread. I
don't think you need to worry about the amount of memory used.
In 0.5.0 we've done some performance improvements that are relevant to you.
Sent from my phone.
Cheers,
-- Andrei
On Jun 10, 2011 8:38 PM, "John Conwell" <jo...@iamjohn.me> wrote:
> In 0.4.0, is Service.launchCluster(ClusterSpec) thread safe? Through the
> API I'm spinning up 3 clusters, one after another, and I'd like to change
> the code to launch them in parallel, but wanted to check if this is
> threadsafe.
>
> --
>
> Thanks,
> John C