Possibilities to setup runner instances on Chop

[Salih Kardan <sa...@gmail.com>]

Hello everyone,

Chop uses AWS EC2 instances to run both chop webapp and runners (instances
which run tests).
We tried hard to make everything as easy as possible in Chop and currently
we are providing an AMI
for runners on which Oracle Java is already installed. However it is not
legal to distribute Oracle Java
in such a public platform, so we had a discussion in our chop team. Here
are the different options
that came out from that discussion, please feel free to comment.


(1) We may provide a public AMI with OpenJDK, actually we already created
one for runner instances.
But, although our runner core runs properly with OpenJDK, code that's going
to be Chop tested may
have problems with it, so it's at users risk to go this way.

(2) We may ask user to provide a link on chop web UI to download oracle tar
ball if he/she wants Oracle JDK
on runner instances. After we download tar ball using that link, we will
install Oracle Java to runner instances.
This way user will have already dealt with license approval process. (User
may provide S3 bucket address for this)

(3) We may ask user to approve the Oracle license agreement with a popup
window while he/she configures
runner instances on web UI, then we can download tar ball from Oracle's
website using wget as explained in
the below link and install it on runner instances.
http://stackoverflow.com/questions/10268583/how-to-automate-download-and-installation-of-java-jdk-on-linux
.

(4) Option #2 and #3 may take a lot time, since on each runner instance
creation, it'll download the tar ball either from link
that user provide or Oracle's website, extract it, configure and install it.
Therefore we came up with option #4 :

Chop web UI may have a separate tool, or an option on UI, so that if the
user wants his private AMI with Oracle JDK
installed inside he can again accept the license agreement, then we open up
an EC2 instance, download & install the
Oracle JDK than make that instance a private AMI for the user, and put its
AMI id in image ID section in the UI.
This way, users can have their private AMIs with just a couple of clicks
and the setup time will not be affected.

Among these options, #1 is ready to use and other options are possible
candidates to be implemented if we agree on them.
Out of these options, of course user can create its own private AMI, and
provide its AMI id to chop web UI to start using Chop.

Regards,
Salih

Re: Possibilities to setup runner instances on Chop

[Dave <sn...@gmail.com>]

Hi Salih,

Thanks for getting the runner script feature in place so quickly, and for
thinking through the options you list below. I'll take a look at that PR as
soon as I can.


On Thu, Jul 24, 2014 at 4:10 AM, Salih Kardan <sa...@gmail.com> wrote:

> Hello everyone,
>
> Chop uses AWS EC2 instances to run both chop webapp and runners (instances
> which run tests).
> We tried hard to make everything as easy as possible in Chop and currently
> we are providing an AMI
> for runners on which Oracle Java is already installed. However it is not
> legal to distribute Oracle Java
> in such a public platform, so we had a discussion in our chop team. Here
> are the different options
> that came out from that discussion, please feel free to comment.
>
> (1) We may provide a public AMI with OpenJDK, actually we already created
> one for runner instances.
> But, although our runner core runs properly with OpenJDK, code that's going
> to be Chop tested may
> have problems with it, so it's at users risk to go this way.
>

That will be very useful for some users of Chop, but for Usergrid we really
need the Oracle JDK because Cassandra still requires it.



> (2) We may ask user to provide a link on chop web UI to download oracle tar
> ball if he/she wants Oracle JDK
> on runner instances. After we download tar ball using that link, we will
> install Oracle Java to runner instances.
> This way user will have already dealt with license approval process. (User
> may provide S3 bucket address for this)
>
> (3) We may ask user to approve the Oracle license agreement with a popup
> window while he/she configures
> runner instances on web UI, then we can download tar ball from Oracle's
> website using wget as explained in
> the below link and install it on runner instances.
>
> http://stackoverflow.com/questions/10268583/how-to-automate-download-and-installation-of-java-jdk-on-linux
> .
>
> (4) Option #2 and #3 may take a lot time, since on each runner instance
> creation, it'll download the tar ball either from link
> that user provide or Oracle's website, extract it, configure and install
> it.
> Therefore we came up with option #4 :
>

Now that we have a runner script, we can install the Oracle JDK by whatever
means we want. I'm not sure that Chop needs to have built-in support for
for Oracle JDK.



> Chop web UI may have a separate tool, or an option on UI, so that if the
> user wants his private AMI with Oracle JDK
> installed inside he can again accept the license agreement, then we open up
> an EC2 instance, download & install the
> Oracle JDK than make that instance a private AMI for the user, and put its
> AMI id in image ID section in the UI.
> This way, users can have their private AMIs with just a couple of clicks
> and the setup time will not be affected.
>

Yes, this is good idea. Chop users should create their own private AMIs
with things like Oracle JDK pre-installed.  However, the process for
creating Chop AMIs is not completely documented (see also
https://issues.apache.org/jira/browse/USERGRID-202)



> Among these options, #1 is ready to use and other options are possible
> candidates to be implemented if we agree on them.
> Out of these options, of course user can create its own private AMI, and
> provide its AMI id to chop web UI to start using Chop.
>


Thanks,
- Dave

Re: Possibilities to setup runner instances on Chop

[Dave <sn...@gmail.com>]

On Thu, Jul 24, 2014 at 8:35 AM, Alex Karasulu <ak...@apache.org> wrote:

> Salih,
>
> Thanks for this very thorough report back to the mailing list summarizing
> off list chop discussions. I wish every project at Apache followed this
> procedure as well as we do here at Usergrid. Well done!
>
> Right now a custom AMI with the Oracle JDK created by users should float
> our Usergrid testing needs for the short term and for others we have option
> #1.
>
> Dave is this sufficient for now?
>

Yes, but I'd like to create my own AMIs with the Oracle JDK installed.


Option #4 seems ideal: let's go with implementing that mechanism for the
> long term if others agree as well.
>


I'm sorry, I mis-read option #4. Now that I have re-read it, I think it is
too complex and not really needed in Chop. If people want to use Oracle
JDK, I think they should to either:

a) write setup scripts that can be executed by a Chop cluster or runner
instance to install Oracle JDK (this will be annoyingly slow, but so be
it). We can do this now thanks to Salih's work on runner startup script.

...or...

b) create private AMIs with Oracle JDK pre-installed. I cannot do this now
because the Chop webapp and runner setup instructions are not complete (see
also https://issues.apache.org/jira/browse/USERGRID-202).


I'm not going to be able to work on my Chop setup again for another week or
more, but I think my next step is #2 to create some Oracle JDK AMIs for my
own private use with Chop.

Thanks again for the help Salih and Alex.

- Dave




>
>

Re: Possibilities to setup runner instances on Chop

[Alex Karasulu <ak...@apache.org>]

Salih,

Thanks for this very thorough report back to the mailing list summarizing
off list chop discussions. I wish every project at Apache followed this
procedure as well as we do here at Usergrid. Well done!

Right now a custom AMI with the Oracle JDK created by users should float
our Usergrid testing needs for the short term and for others we have option
#1.

Dave is this sufficient for now?

Option #4 seems ideal: let's go with implementing that mechanism for the
long term if others agree as well.

Thoughts?

Thanks,
Alex



On Thu, Jul 24, 2014 at 2:10 PM, Salih Kardan <sa...@gmail.com> wrote:

> Hello everyone,
>
> Chop uses AWS EC2 instances to run both chop webapp and runners (instances
> which run tests).
> We tried hard to make everything as easy as possible in Chop and currently
> we are providing an AMI
> for runners on which Oracle Java is already installed. However it is not
> legal to distribute Oracle Java
> in such a public platform, so we had a discussion in our chop team. Here
> are the different options
> that came out from that discussion, please feel free to comment.
>
>
> (1) We may provide a public AMI with OpenJDK, actually we already created
> one for runner instances.
> But, although our runner core runs properly with OpenJDK, code that's going
> to be Chop tested may
> have problems with it, so it's at users risk to go this way.
>
> (2) We may ask user to provide a link on chop web UI to download oracle tar
> ball if he/she wants Oracle JDK
> on runner instances. After we download tar ball using that link, we will
> install Oracle Java to runner instances.
> This way user will have already dealt with license approval process. (User
> may provide S3 bucket address for this)
>
> (3) We may ask user to approve the Oracle license agreement with a popup
> window while he/she configures
> runner instances on web UI, then we can download tar ball from Oracle's
> website using wget as explained in
> the below link and install it on runner instances.
>
> http://stackoverflow.com/questions/10268583/how-to-automate-download-and-installation-of-java-jdk-on-linux
> .
>
> (4) Option #2 and #3 may take a lot time, since on each runner instance
> creation, it'll download the tar ball either from link
> that user provide or Oracle's website, extract it, configure and install
> it.
> Therefore we came up with option #4 :
>
> Chop web UI may have a separate tool, or an option on UI, so that if the
> user wants his private AMI with Oracle JDK
> installed inside he can again accept the license agreement, then we open up
> an EC2 instance, download & install the
> Oracle JDK than make that instance a private AMI for the user, and put its
> AMI id in image ID section in the UI.
> This way, users can have their private AMIs with just a couple of clicks
> and the setup time will not be affected.
>
> Among these options, #1 is ready to use and other options are possible
> candidates to be implemented if we agree on them.
> Out of these options, of course user can create its own private AMI, and
> provide its AMI id to chop web UI to start using Chop.
>
> Regards,
> Salih
>



-- 
Best Regards,
-- Alex