You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@cloudstack.apache.org by Nathan Johnson <nj...@ena.com> on 2017/02/24 16:25:22 UTC

Modern template hosting

So not to re-open a can of worms, but I’m in a situation where I need to  
come up with a Marvin component test that depends on a template based on a  
kernel that’s relatively new, i.e., newer than Centos 5.3 / Ubuntu 10.04 .   
I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but from  
looking at the thread "Migrating CloudStack content from  
download.cloud.com” it looks like there is resistance to using this at  
least for hosting system vm templates over concerns of neutrality.  Would  
this be suitable for a component test?  If not, what is a “blessed”  
template location?

Thanks in advance!

Nathan Johnson
R&D Engineer



618 Grassmere Park Drive, Suite 12
Nashville, TN 37211
General Office: 615-312-6000

website | blog | support

Re: Modern template hosting

Posted by Nathan Johnson <nj...@ena.com>.

Paul Angus <pa...@shapeblue.com> wrote:

> Hi Nathan,
>
> Ideally, if you put the template location in (or use a template defined  
> in)  test_data.py then the actual location can be overridden by anyone  
> testing.
>
> For Trillian, we've copied all of the templates that people have define  
> to a local repo and then replace the URLs in test_data.py to reduce  
> bandwidth use and download times.
>
> Ie:
>
>             "bootableIso":
>                 {
>                     "displaytext": "Test Bootable ISO",
>                     "name": "testISO",
>                     "bootable": True,
>                     "ispublic": False,
>                     "url": "{{ marvin_images_location }}/TinyCore-current.iso",
>                     "ostype": 'Other Linux (64-bit)',
>                     "mode": 'HTTP_DOWNLOAD'
>         },
>


Thanks for the tip, this could work pretty well for us I think.

Re: Modern template hosting

Posted by Raja Pullela <ra...@accelerite.com>.

sorry Wido… forgot… thank you for correcting!

On 2/27/17, 4:50 PM, "Wido den Hollander" <wi...@widodh.nl> wrote:
> Op 27 februari 2017 om 7:17 schreef Raja Pullela <ra...@accelerite.com>:
> 
> 
> Hi will, 
> 
> I believe, we didn’t get to close ‘getting a mirror on Apache’ because we needed someone on the Apache Infra side to close this. BTW, cloudstack-apt.get.eu (I think Nux manages this?) has all/most of the content.  Once we can close on the Apache mirror for hosting the content, I can help assist getting the content there.
> 

cloudstack.apt-get.eu is managed by me.

> For now, we have replicated the download.cloud.com content to ‘s3.download.accelerite.com’.
> Also, we are working on a set of steps/procedure to help with this change.  I will update everyone in about a week’s time on the details.
> 

Thanks! Although we should probably refrain for using a non-generic domain name in any of the source.

Wido

> Best,
> Raja Pullela
> Engineering Team,
> Accelerite, 2055 Laurelwood Road,
> Santa Clara, CA, 95054
> 
> On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of Will Stevens" <williamstevens@gmail.com on behalf of wstevens@cloudops.com> wrote:
> 
> unfortunately the template mirror conversation got caught up in details and
> nobody took the lead on implementing a solution.
> 
> citrix has been pinging me every couple months to say 'dude, we need to
> remove the dependency on download.citrix.com', but i have not had the
> cycles to get in and solve the problem.  the shutdown of that is imminent
> right now, so we need to solve it asap.
> 
> 
> 
> *Will STEVENS*
> Lead Developer
> 
> <https://goo.gl/NYZ8KK>
> 
> On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <pa...@shapeblue.com>
> wrote:
> 
> > Hi Nathan,
> >
> > Ideally, if you put the template location in (or use a template defined
> > in)  test_data.py then the actual location can be overridden by anyone
> > testing.
> >
> > For Trillian, we've copied all of the templates that people have define to
> > a local repo and then replace the URLs in test_data.py to reduce bandwidth
> > use and download times.
> >
> > Ie:
> >
> >             "bootableIso":
> >                 {
> >                     "displaytext": "Test Bootable ISO",
> >                     "name": "testISO",
> >                     "bootable": True,
> >                     "ispublic": False,
> >                     "url": "{{ marvin_images_location
> > }}/TinyCore-current.iso",
> >                     "ostype": 'Other Linux (64-bit)',
> >                     "mode": 'HTTP_DOWNLOAD'
> >         },
> >
> >
> >
> > I thought that we had come up with a solution for download.cloud.com, by
> > having a mirrorlist hosted in Community Apache 'space' with anyone able to
> > out themselves forward as a mirror.
> > But I must admit I lost track of whether anyone made the requisite changes
> > in code....
> >
> >
> >
> >
> > Kind regards,
> >
> > Paul Angus
> >
> > paul.angus@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> > -----Original Message-----
> > From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
> > Behalf Of Will Stevens
> > Sent: 24 February 2017 16:30
> > To: dev@cloudstack.apache.org
> > Subject: Re: Modern template hosting
> >
> > this is a hard questions.  in general, we should be setting up a mirror on
> > some cloudstack/apache domain and then mirror to other provided templates.
> >
> > we MUST come up with a solution to deprecate 'download.cloud.com', that
> > is going to be going away any day now.
> >
> > i don't know the right way to solve this to be honest, but if you have
> > ideas, i am willing to help.
> >
> >
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <nj...@ena.com> wrote:
> >
> > > So not to re-open a can of worms, but I’m in a situation where I need
> > > to come up with a Marvin component test that depends on a template
> > > based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
> > Ubuntu 10.04 .
> > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
> > > from looking at the thread "Migrating CloudStack content from
> > > download.cloud.com” it looks like there is resistance to using this at
> > > least for hosting system vm templates over concerns of neutrality.
> > > Would this be suitable for a component test?  If not, what is a “blessed”
> > > template location?
> > >
> > > Thanks in advance!
> > >
> > > Nathan Johnson
> > > R&D Engineer
> > >
> > >
> > >
> > > 618 Grassmere Park Drive, Suite 12
> > > Nashville, TN 37211
> > > General Office: 615-312-6000
> > >
> > > website | blog | support
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> 
> 
> 
> 
> 
> 
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.





DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.

Re: Modern template hosting

Posted by Wido den Hollander <wi...@widodh.nl>.

> Op 27 februari 2017 om 7:17 schreef Raja Pullela <ra...@accelerite.com>:
> 
> 
> Hi will, 
> 
> I believe, we didn’t get to close ‘getting a mirror on Apache’ because we needed someone on the Apache Infra side to close this. BTW, cloudstack-apt.get.eu (I think Nux manages this?) has all/most of the content.  Once we can close on the Apache mirror for hosting the content, I can help assist getting the content there.
> 

cloudstack.apt-get.eu is managed by me.

> For now, we have replicated the download.cloud.com content to ‘s3.download.accelerite.com’.
> Also, we are working on a set of steps/procedure to help with this change.  I will update everyone in about a week’s time on the details.
> 

Thanks! Although we should probably refrain for using a non-generic domain name in any of the source.

Wido

> Best,
> Raja Pullela
> Engineering Team,
> Accelerite, 2055 Laurelwood Road,
> Santa Clara, CA, 95054
> 
> On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of Will Stevens" <williamstevens@gmail.com on behalf of wstevens@cloudops.com> wrote:
> 
> unfortunately the template mirror conversation got caught up in details and
> nobody took the lead on implementing a solution.
> 
> citrix has been pinging me every couple months to say 'dude, we need to
> remove the dependency on download.citrix.com', but i have not had the
> cycles to get in and solve the problem.  the shutdown of that is imminent
> right now, so we need to solve it asap.
> 
> 
> 
> *Will STEVENS*
> Lead Developer
> 
> <https://goo.gl/NYZ8KK>
> 
> On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <pa...@shapeblue.com>
> wrote:
> 
> > Hi Nathan,
> >
> > Ideally, if you put the template location in (or use a template defined
> > in)  test_data.py then the actual location can be overridden by anyone
> > testing.
> >
> > For Trillian, we've copied all of the templates that people have define to
> > a local repo and then replace the URLs in test_data.py to reduce bandwidth
> > use and download times.
> >
> > Ie:
> >
> >             "bootableIso":
> >                 {
> >                     "displaytext": "Test Bootable ISO",
> >                     "name": "testISO",
> >                     "bootable": True,
> >                     "ispublic": False,
> >                     "url": "{{ marvin_images_location
> > }}/TinyCore-current.iso",
> >                     "ostype": 'Other Linux (64-bit)',
> >                     "mode": 'HTTP_DOWNLOAD'
> >         },
> >
> >
> >
> > I thought that we had come up with a solution for download.cloud.com, by
> > having a mirrorlist hosted in Community Apache 'space' with anyone able to
> > out themselves forward as a mirror.
> > But I must admit I lost track of whether anyone made the requisite changes
> > in code....
> >
> >
> >
> >
> > Kind regards,
> >
> > Paul Angus
> >
> > paul.angus@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> > -----Original Message-----
> > From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
> > Behalf Of Will Stevens
> > Sent: 24 February 2017 16:30
> > To: dev@cloudstack.apache.org
> > Subject: Re: Modern template hosting
> >
> > this is a hard questions.  in general, we should be setting up a mirror on
> > some cloudstack/apache domain and then mirror to other provided templates.
> >
> > we MUST come up with a solution to deprecate 'download.cloud.com', that
> > is going to be going away any day now.
> >
> > i don't know the right way to solve this to be honest, but if you have
> > ideas, i am willing to help.
> >
> >
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <nj...@ena.com> wrote:
> >
> > > So not to re-open a can of worms, but I’m in a situation where I need
> > > to come up with a Marvin component test that depends on a template
> > > based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
> > Ubuntu 10.04 .
> > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
> > > from looking at the thread "Migrating CloudStack content from
> > > download.cloud.com” it looks like there is resistance to using this at
> > > least for hosting system vm templates over concerns of neutrality.
> > > Would this be suitable for a component test?  If not, what is a “blessed”
> > > template location?
> > >
> > > Thanks in advance!
> > >
> > > Nathan Johnson
> > > R&D Engineer
> > >
> > >
> > >
> > > 618 Grassmere Park Drive, Suite 12
> > > Nashville, TN 37211
> > > General Office: 615-312-6000
> > >
> > > website | blog | support
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> 
> 
> 
> 
> 
> 
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.

Re: Modern template hosting

Posted by Nux! <nu...@li.nux.ro>.

Raja,

apt-get.eu is Wido's.

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Raja Pullela" <ra...@accelerite.com>
> To: "dev" <de...@cloudstack.apache.org>
> Sent: Monday, 27 February, 2017 06:17:36
> Subject: Re: Modern template hosting

> Hi will,
> 
> I believe, we didn’t get to close ‘getting a mirror on Apache’ because we needed
> someone on the Apache Infra side to close this. BTW, cloudstack-apt.get.eu (I
> think Nux manages this?) has all/most of the content.  Once we can close on the
> Apache mirror for hosting the content, I can help assist getting the content
> there.
> 
> For now, we have replicated the download.cloud.com content to
> ‘s3.download.accelerite.com’.
> Also, we are working on a set of steps/procedure to help with this change.  I
> will update everyone in about a week’s time on the details.
> 
> Best,
> Raja Pullela
> Engineering Team,
> Accelerite, 2055 Laurelwood Road,
> Santa Clara, CA, 95054
> 
> On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of Will Stevens"
> <williamstevens@gmail.com on behalf of wstevens@cloudops.com> wrote:
> 
> unfortunately the template mirror conversation got caught up in details and
> nobody took the lead on implementing a solution.
> 
> citrix has been pinging me every couple months to say 'dude, we need to
> remove the dependency on download.citrix.com', but i have not had the
> cycles to get in and solve the problem.  the shutdown of that is imminent
> right now, so we need to solve it asap.
> 
> 
> 
> *Will STEVENS*
> Lead Developer
> 
> <https://goo.gl/NYZ8KK>
> 
> On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <pa...@shapeblue.com>
> wrote:
> 
>> Hi Nathan,
>>
>> Ideally, if you put the template location in (or use a template defined
>> in)  test_data.py then the actual location can be overridden by anyone
>> testing.
>>
>> For Trillian, we've copied all of the templates that people have define to
>> a local repo and then replace the URLs in test_data.py to reduce bandwidth
>> use and download times.
>>
>> Ie:
>>
>>             "bootableIso":
>>                 {
>>                     "displaytext": "Test Bootable ISO",
>>                     "name": "testISO",
>>                     "bootable": True,
>>                     "ispublic": False,
>>                     "url": "{{ marvin_images_location
>> }}/TinyCore-current.iso",
>>                     "ostype": 'Other Linux (64-bit)',
>>                     "mode": 'HTTP_DOWNLOAD'
>>         },
>>
>>
>>
>> I thought that we had come up with a solution for download.cloud.com, by
>> having a mirrorlist hosted in Community Apache 'space' with anyone able to
>> out themselves forward as a mirror.
>> But I must admit I lost track of whether anyone made the requisite changes
>> in code....
>>
>>
>>
>>
>> Kind regards,
>>
>> Paul Angus
>>
>> paul.angus@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> @shapeblue
>>
>>
>>
>>
>> -----Original Message-----
>> From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
>> Behalf Of Will Stevens
>> Sent: 24 February 2017 16:30
>> To: dev@cloudstack.apache.org
>> Subject: Re: Modern template hosting
>>
>> this is a hard questions.  in general, we should be setting up a mirror on
>> some cloudstack/apache domain and then mirror to other provided templates.
>>
>> we MUST come up with a solution to deprecate 'download.cloud.com', that
>> is going to be going away any day now.
>>
>> i don't know the right way to solve this to be honest, but if you have
>> ideas, i am willing to help.
>>
>>
>>
>> *Will STEVENS*
>> Lead Developer
>>
>> <https://goo.gl/NYZ8KK>
>>
>> On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <nj...@ena.com> wrote:
>>
>> > So not to re-open a can of worms, but I’m in a situation where I need
>> > to come up with a Marvin component test that depends on a template
>> > based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
>> Ubuntu 10.04 .
>> > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
>> > from looking at the thread "Migrating CloudStack content from
>> > download.cloud.com” it looks like there is resistance to using this at
>> > least for hosting system vm templates over concerns of neutrality.
>> > Would this be suitable for a component test?  If not, what is a “blessed”
>> > template location?
>> >
>> > Thanks in advance!
>> >
>> > Nathan Johnson
>> > R&D Engineer
>> >
>> >
>> >
>> > 618 Grassmere Park Drive, Suite 12
>> > Nashville, TN 37211
>> > General Office: 615-312-6000
>> >
>> > website | blog | support
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>>
> 
> 
> 
> 
> 
> 
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the
> property of Accelerite, a Persistent Systems business. It is intended only for
> the use of the individual or entity to which it is addressed. If you are not
> the intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Accelerite, a Persistent Systems business does not accept any liability for
> virus infected mails.

Re: Modern template hosting

Posted by Chiradeep Vittal <ch...@gmail.com>.

Infra supports Ubuntu VMs apparently so you could do what you want there
https://www.apache.org/dev/services#virtual-servers

Another idea is to distribute the http server binary with ACS - the
CloudStack admin installs the http server and gives it a reachable fqdn /
ip and updates the templates table.


On Mon, Feb 27, 2017 at 12:48 PM, Will Stevens <ws...@cloudops.com>
wrote:

> so this is what I am looking to do.  Please let me know if you have
> suggestions for me or think I should be solving the problem a different
> way.
>
> - We request a new Github repository from the ASF at:
> 'apache/cloudstack-mirror-list'
> - In this repository we track a text file in the 'gh-pages' branch with a
> list of valid download mirrors.
> - I build a binary to be hosted by the ASF (or at least with the ASF
> pointing a domain at the binary and I could potentially host it).  We will
> see how they want to handle the hosting of the binary.
>
> The binary would expose a web server which would behave as follows:
> - When the 'client' requests a download url the following flow is kicked
> off:
> -- The mirror list is queried from github (or from a static site hosted on
> asf, as we see fit).
> -- The Lat/Lon of the 'client' is determined based on their IP.
> -- The Lat/Lon for each of the 'mirror's is determined based on an IP
> lookup of the hostname.
> -- The closest geographical mirror is determined, the target is validated
> to be available and the user is redirected.
>
> Some questions I have right now:
> - Will every mirror have the same path structure to access the equivalent
> resources?
> - Should we support adding a path to the mirror url to specify the path to
> the base common path?
> -- Example: lets say the binary is hosted on 'dl.acs.com' and there are
> three mirrors 'abc.com', 'pqr.com/files' and 'xyx.com/downloads'.
> -- If the path being requested is '
> dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would result in the
> following potential paths for the mirrors:
> -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> -- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'
>
> Does this all make sense?
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <ch...@gmail.com>
> wrote:
>
> > My bad. A few lines down, this has been added recently:
> >
> > this.request.setFollowRedirects(true);
> >
> > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <williamstevens@gmail.com
> >
> > wrote:
> >
> > > OK. Thanks for the heads up.
> > >
> > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <ch...@gmail.com>
> > wrote:
> > >
> > > > Sounds workable. The downloader code in the SSVM won't follow
> > redirects I
> > > > think.
> > > > https://github.com/apache/cloudstack/blob/
> > 5511065fc20787619d9cd0444a65a3
> > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > HttpTemplateDownloader.java#L93
> > > > https://goo.gl/dSi0r5
> > > >  Might need to add
> > > > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > >
> > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <wstevens@cloudops.com
> >
> > > > wrote:
> > > >
> > > > > We haven't opened a ticket yet because we don't have a strategy
> yet.
> > > > >
> > > > > What do you guys think of this:
> > > > > - We setup a new github repo in the 'apache' org which consists of
> a
> > > > single
> > > > > file with a list of active/supported mirrors.
> > > > > - I write a small web server, distributed as a binary, which can be
> > > > hosted
> > > > > by ASF Infra.  This web server will query the current list of
> mirrors
> > > and
> > > > > will select one and then do a 302 redirect to that mirror.
> > > > >
> > > > > The act of 'choosing' a mirror could be done in a number of ways.
> > > > > - If we want to define an order, then it could just try from the
> top
> > of
> > > > the
> > > > > list and work its way down.  It would curl the target to make sure
> it
> > > > gets
> > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > - Or, if we want to distribute the load across the mirrors, we
> could
> > > pick
> > > > > from the list randomly.  Again, doing a curl to verify the mirror
> is
> > up
> > > > and
> > > > > then doing a redirect.
> > > > > - If we want to get fancy, we could do a reverse IP lookup and try
> to
> > > > match
> > > > > the requester with their closest geographical mirror.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > *Will STEVENS*
> > > > > Lead Developer
> > > > >
> > > > > <https://goo.gl/NYZ8KK>
> > > > >
> > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > chiradeepv@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > What steps are needed to set up a mirror? What does Infra need to
> > do?
> > > > Has
> > > > > > anybody filed a ticket with Infra?
> > > > > >
> > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > raja.pullela@accelerite.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi will,
> > > > > > >
> > > > > > > I believe, we didn’t get to close ‘getting a mirror on Apache’
> > > > because
> > > > > we
> > > > > > > needed someone on the Apache Infra side to close this. BTW,
> > > > > > > cloudstack-apt.get.eu (I think Nux manages this?) has all/most
> > of
> > > > the
> > > > > > > content.  Once we can close on the Apache mirror for hosting
> the
> > > > > > content, I
> > > > > > > can help assist getting the content there.
> > > > > > >
> > > > > > > For now, we have replicated the download.cloud.com content to
> ‘
> > > > > > > s3.download.accelerite.com’.
> > > > > > > Also, we are working on a set of steps/procedure to help with
> > this
> > > > > > > change.  I will update everyone in about a week’s time on the
> > > > details.
> > > > > > >
> > > > > > > Best,
> > > > > > > Raja Pullela
> > > > > > > Engineering Team,
> > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > Santa Clara, CA, 95054
> > > > > > >
> > > > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of
> > Will
> > > > > > > Stevens" <williamstevens@gmail.com on behalf of
> > > > wstevens@cloudops.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > unfortunately the template mirror conversation got caught up in
> > > > details
> > > > > > and
> > > > > > > nobody took the lead on implementing a solution.
> > > > > > >
> > > > > > > citrix has been pinging me every couple months to say 'dude, we
> > > need
> > > > to
> > > > > > > remove the dependency on download.citrix.com', but i have not
> > had
> > > > the
> > > > > > > cycles to get in and solve the problem.  the shutdown of that
> is
> > > > > imminent
> > > > > > > right now, so we need to solve it asap.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *Will STEVENS*
> > > > > > > Lead Developer
> > > > > > >
> > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > >
> > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > paul.angus@shapeblue.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Nathan,
> > > > > > > >
> > > > > > > > Ideally, if you put the template location in (or use a
> template
> > > > > defined
> > > > > > > > in)  test_data.py then the actual location can be overridden
> by
> > > > > anyone
> > > > > > > > testing.
> > > > > > > >
> > > > > > > > For Trillian, we've copied all of the templates that people
> > have
> > > > > define
> > > > > > > to
> > > > > > > > a local repo and then replace the URLs in test_data.py to
> > reduce
> > > > > > > bandwidth
> > > > > > > > use and download times.
> > > > > > > >
> > > > > > > > Ie:
> > > > > > > >
> > > > > > > >             "bootableIso":
> > > > > > > >                 {
> > > > > > > >                     "displaytext": "Test Bootable ISO",
> > > > > > > >                     "name": "testISO",
> > > > > > > >                     "bootable": True,
> > > > > > > >                     "ispublic": False,
> > > > > > > >                     "url": "{{ marvin_images_location
> > > > > > > > }}/TinyCore-current.iso",
> > > > > > > >                     "ostype": 'Other Linux (64-bit)',
> > > > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > > > >         },
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > I thought that we had come up with a solution for
> > > > download.cloud.com
> > > > > ,
> > > > > > by
> > > > > > > > having a mirrorlist hosted in Community Apache 'space' with
> > > anyone
> > > > > able
> > > > > > > to
> > > > > > > > out themselves forward as a mirror.
> > > > > > > > But I must admit I lost track of whether anyone made the
> > > requisite
> > > > > > > changes
> > > > > > > > in code....
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Kind regards,
> > > > > > > >
> > > > > > > > Paul Angus
> > > > > > > >
> > > > > > > > paul.angus@shapeblue.com
> > > > > > > > www.shapeblue.com
> > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > > @shapeblue
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: williamstevens@gmail.com [mailto:williamstevens@gmail.
> > com]
> > > > On
> > > > > > > > Behalf Of Will Stevens
> > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > Subject: Re: Modern template hosting
> > > > > > > >
> > > > > > > > this is a hard questions.  in general, we should be setting
> up
> > a
> > > > > mirror
> > > > > > > on
> > > > > > > > some cloudstack/apache domain and then mirror to other
> provided
> > > > > > > templates.
> > > > > > > >
> > > > > > > > we MUST come up with a solution to deprecate '
> > download.cloud.com
> > > ',
> > > > > > that
> > > > > > > > is going to be going away any day now.
> > > > > > > >
> > > > > > > > i don't know the right way to solve this to be honest, but if
> > you
> > > > > have
> > > > > > > > ideas, i am willing to help.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *Will STEVENS*
> > > > > > > > Lead Developer
> > > > > > > >
> > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > >
> > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> > > njohnson@ena.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > So not to re-open a can of worms, but I’m in a situation
> > where
> > > I
> > > > > need
> > > > > > > > > to come up with a Marvin component test that depends on a
> > > > template
> > > > > > > > > based on a kernel that’s relatively new, i.e., newer than
> > > Centos
> > > > > 5.3
> > > > > > /
> > > > > > > > Ubuntu 10.04 .
> > > > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for
> > > KVM),
> > > > > but
> > > > > > > > > from looking at the thread "Migrating CloudStack content
> from
> > > > > > > > > download.cloud.com” it looks like there is resistance to
> > using
> > > > > this
> > > > > > at
> > > > > > > > > least for hosting system vm templates over concerns of
> > > > neutrality.
> > > > > > > > > Would this be suitable for a component test?  If not, what
> > is a
> > > > > > > “blessed”
> > > > > > > > > template location?
> > > > > > > > >
> > > > > > > > > Thanks in advance!
> > > > > > > > >
> > > > > > > > > Nathan Johnson
> > > > > > > > > R&D Engineer
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > Nashville, TN 37211
> > > > > > > > > General Office: 615-312-6000
> > > > > > > > >
> > > > > > > > > website | blog | support
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > DISCLAIMER
> > > > > > > ==========
> > > > > > > This e-mail may contain privileged and confidential information
> > > which
> > > > > is
> > > > > > > the property of Accelerite, a Persistent Systems business. It
> is
> > > > > intended
> > > > > > > only for the use of the individual or entity to which it is
> > > > addressed.
> > > > > If
> > > > > > > you are not the intended recipient, you are not authorized to
> > read,
> > > > > > retain,
> > > > > > > copy, print, distribute or use this message. If you have
> received
> > > > this
> > > > > > > communication in error, please notify the sender and delete all
> > > > copies
> > > > > of
> > > > > > > this message. Accelerite, a Persistent Systems business does
> not
> > > > accept
> > > > > > any
> > > > > > > liability for virus infected mails.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Modern template hosting

Posted by Rafael Weingärtner <ra...@gmail.com>.

Another thing, in my opinion, we should avoid these hashes (MD5, SHA1, and
others) where hash collisions have already been found; if we will be
working to create a new working flow for managing these system VMs
templates, I think we should instead use SHA256 or SHA512.

On Mon, Feb 27, 2017 at 4:55 PM, Will Stevens <ws...@cloudops.com> wrote:

> @rafael: Yes, I think it makes sense to support sub-paths.  It will make
> things a lot easier to work with.  The security question is an interesting
> one.  I am not sure I would be the best person to answer that.  I would say
> that we would want to have the MD5s hosted in the same repo as the mirror
> list urls.
>
> I do like the idea of running a web server on the management server for
> hosting the system vms though.  It could be used to populate secondary
> storage and could potentially give operators an easier (more efficient?)
> workflow for introducing new templates.  That area could use some work...
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Mon, Feb 27, 2017 at 4:36 PM, Chiradeep Vittal <ch...@gmail.com>
> wrote:
>
> > Hashes are checked (md5 IIRC) today.
> > But given the issues, I think the project should steer away from hosting
> > templates except the systemvm template.
> >
> > On Mon, Feb 27, 2017 at 1:31 PM, Rafael Weingärtner <
> > rafaelweingartner@gmail.com> wrote:
> >
> > > Will, I think we could support different path structures. This can
> > > facilitate different deployment of mirrors based on the structure the
> > host
> > > has.
> > >
> > > Could I add something else to the discussion? Have we discussed the
> > > security impacts of setting up this mirrors approach?
> > > I mean, if any of the mirrors gets corrupted (let`s say by a hacker),
> and
> > > the templates are injected with malicious code, an attacker could
> > > potentially get un-monitored and unlimited access to a cloud
> environment.
> > >
> > > If we assume that the mirror may get malicious (it is not that I do not
> > > trust you guys, but bad things happen), we cannot host hashes there.
> > Where
> > > do you think we could store Sha512 or another hash type for these
> > > templates? Could we host in the newly proposed Github repo or maybe
> some
> > > place in the ACS website?
> > >
> > > This would have an impact on clients (needing clear documentation) and
> > our
> > > code that automatically downloads System VM templates (does it check
> > hashes
> > > when automatically installing templates today? It may require
> > > implementation changes).
> > >
> > > On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <ws...@cloudops.com>
> > > wrote:
> > >
> > > > so this is what I am looking to do.  Please let me know if you have
> > > > suggestions for me or think I should be solving the problem a
> different
> > > > way.
> > > >
> > > > - We request a new Github repository from the ASF at:
> > > > 'apache/cloudstack-mirror-list'
> > > > - In this repository we track a text file in the 'gh-pages' branch
> > with a
> > > > list of valid download mirrors.
> > > > - I build a binary to be hosted by the ASF (or at least with the ASF
> > > > pointing a domain at the binary and I could potentially host it).  We
> > > will
> > > > see how they want to handle the hosting of the binary.
> > > >
> > > > The binary would expose a web server which would behave as follows:
> > > > - When the 'client' requests a download url the following flow is
> > kicked
> > > > off:
> > > > -- The mirror list is queried from github (or from a static site
> hosted
> > > on
> > > > asf, as we see fit).
> > > > -- The Lat/Lon of the 'client' is determined based on their IP.
> > > > -- The Lat/Lon for each of the 'mirror's is determined based on an IP
> > > > lookup of the hostname.
> > > > -- The closest geographical mirror is determined, the target is
> > validated
> > > > to be available and the user is redirected.
> > > >
> > > > Some questions I have right now:
> > > > - Will every mirror have the same path structure to access the
> > equivalent
> > > > resources?
> > > > - Should we support adding a path to the mirror url to specify the
> path
> > > to
> > > > the base common path?
> > > > -- Example: lets say the binary is hosted on 'dl.acs.com' and there
> > are
> > > > three mirrors 'abc.com', 'pqr.com/files' and 'xyx.com/downloads'.
> > > > -- If the path being requested is '
> > > > dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would result in
> the
> > > > following potential paths for the mirrors:
> > > > -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> > > > -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> > > > -- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'
> > > >
> > > > Does this all make sense?
> > > >
> > > > *Will STEVENS*
> > > > Lead Developer
> > > >
> > > > <https://goo.gl/NYZ8KK>
> > > >
> > > > On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <
> > chiradeepv@gmail.com>
> > > > wrote:
> > > >
> > > > > My bad. A few lines down, this has been added recently:
> > > > >
> > > > > this.request.setFollowRedirects(true);
> > > > >
> > > > > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <
> > > williamstevens@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > OK. Thanks for the heads up.
> > > > > >
> > > > > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> chiradeepv@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Sounds workable. The downloader code in the SSVM won't follow
> > > > > redirects I
> > > > > > > think.
> > > > > > > https://github.com/apache/cloudstack/blob/
> > > > > 5511065fc20787619d9cd0444a65a3
> > > > > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > > > > HttpTemplateDownloader.java#L93
> > > > > > > https://goo.gl/dSi0r5
> > > > > > >  Might need to add
> > > > > > > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > > > > >
> > > > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <
> > > wstevens@cloudops.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > We haven't opened a ticket yet because we don't have a
> strategy
> > > > yet.
> > > > > > > >
> > > > > > > > What do you guys think of this:
> > > > > > > > - We setup a new github repo in the 'apache' org which
> consists
> > > of
> > > > a
> > > > > > > single
> > > > > > > > file with a list of active/supported mirrors.
> > > > > > > > - I write a small web server, distributed as a binary, which
> > can
> > > be
> > > > > > > hosted
> > > > > > > > by ASF Infra.  This web server will query the current list of
> > > > mirrors
> > > > > > and
> > > > > > > > will select one and then do a 302 redirect to that mirror.
> > > > > > > >
> > > > > > > > The act of 'choosing' a mirror could be done in a number of
> > ways.
> > > > > > > > - If we want to define an order, then it could just try from
> > the
> > > > top
> > > > > of
> > > > > > > the
> > > > > > > > list and work its way down.  It would curl the target to make
> > > sure
> > > > it
> > > > > > > gets
> > > > > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > > > > - Or, if we want to distribute the load across the mirrors,
> we
> > > > could
> > > > > > pick
> > > > > > > > from the list randomly.  Again, doing a curl to verify the
> > mirror
> > > > is
> > > > > up
> > > > > > > and
> > > > > > > > then doing a redirect.
> > > > > > > > - If we want to get fancy, we could do a reverse IP lookup
> and
> > > try
> > > > to
> > > > > > > match
> > > > > > > > the requester with their closest geographical mirror.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > *Will STEVENS*
> > > > > > > > Lead Developer
> > > > > > > >
> > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > >
> > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > > > chiradeepv@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > What steps are needed to set up a mirror? What does Infra
> > need
> > > to
> > > > > do?
> > > > > > > Has
> > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > >
> > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > > raja.pullela@accelerite.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi will,
> > > > > > > > > >
> > > > > > > > > > I believe, we didn’t get to close ‘getting a mirror on
> > > Apache’
> > > > > > > because
> > > > > > > > we
> > > > > > > > > > needed someone on the Apache Infra side to close this.
> BTW,
> > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
> > > all/most
> > > > > of
> > > > > > > the
> > > > > > > > > > content.  Once we can close on the Apache mirror for
> > hosting
> > > > the
> > > > > > > > > content, I
> > > > > > > > > > can help assist getting the content there.
> > > > > > > > > >
> > > > > > > > > > For now, we have replicated the download.cloud.com
> content
> > > to
> > > > ‘
> > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > Also, we are working on a set of steps/procedure to help
> > with
> > > > > this
> > > > > > > > > > change.  I will update everyone in about a week’s time on
> > the
> > > > > > > details.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Raja Pullela
> > > > > > > > > > Engineering Team,
> > > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > > >
> > > > > > > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on
> behalf
> > of
> > > > > Will
> > > > > > > > > > Stevens" <williamstevens@gmail.com on behalf of
> > > > > > > wstevens@cloudops.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > unfortunately the template mirror conversation got caught
> > up
> > > in
> > > > > > > details
> > > > > > > > > and
> > > > > > > > > > nobody took the lead on implementing a solution.
> > > > > > > > > >
> > > > > > > > > > citrix has been pinging me every couple months to say
> > 'dude,
> > > we
> > > > > > need
> > > > > > > to
> > > > > > > > > > remove the dependency on download.citrix.com', but i
> have
> > > not
> > > > > had
> > > > > > > the
> > > > > > > > > > cycles to get in and solve the problem.  the shutdown of
> > that
> > > > is
> > > > > > > > imminent
> > > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > *Will STEVENS*
> > > > > > > > > > Lead Developer
> > > > > > > > > >
> > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > > > > paul.angus@shapeblue.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Nathan,
> > > > > > > > > > >
> > > > > > > > > > > Ideally, if you put the template location in (or use a
> > > > template
> > > > > > > > defined
> > > > > > > > > > > in)  test_data.py then the actual location can be
> > > overridden
> > > > by
> > > > > > > > anyone
> > > > > > > > > > > testing.
> > > > > > > > > > >
> > > > > > > > > > > For Trillian, we've copied all of the templates that
> > people
> > > > > have
> > > > > > > > define
> > > > > > > > > > to
> > > > > > > > > > > a local repo and then replace the URLs in test_data.py
> to
> > > > > reduce
> > > > > > > > > > bandwidth
> > > > > > > > > > > use and download times.
> > > > > > > > > > >
> > > > > > > > > > > Ie:
> > > > > > > > > > >
> > > > > > > > > > >             "bootableIso":
> > > > > > > > > > >                 {
> > > > > > > > > > >                     "displaytext": "Test Bootable ISO",
> > > > > > > > > > >                     "name": "testISO",
> > > > > > > > > > >                     "bootable": True,
> > > > > > > > > > >                     "ispublic": False,
> > > > > > > > > > >                     "url": "{{ marvin_images_location
> > > > > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > > > >                     "ostype": 'Other Linux (64-bit)',
> > > > > > > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > > > > > > >         },
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I thought that we had come up with a solution for
> > > > > > > download.cloud.com
> > > > > > > > ,
> > > > > > > > > by
> > > > > > > > > > > having a mirrorlist hosted in Community Apache 'space'
> > with
> > > > > > anyone
> > > > > > > > able
> > > > > > > > > > to
> > > > > > > > > > > out themselves forward as a mirror.
> > > > > > > > > > > But I must admit I lost track of whether anyone made
> the
> > > > > > requisite
> > > > > > > > > > changes
> > > > > > > > > > > in code....
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Kind regards,
> > > > > > > > > > >
> > > > > > > > > > > Paul Angus
> > > > > > > > > > >
> > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > www.shapeblue.com
> > > > > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > > > > > @shapeblue
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: williamstevens@gmail.com [mailto:
> > > williamstevens@gmail.
> > > > > com]
> > > > > > > On
> > > > > > > > > > > Behalf Of Will Stevens
> > > > > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > > > Subject: Re: Modern template hosting
> > > > > > > > > > >
> > > > > > > > > > > this is a hard questions.  in general, we should be
> > setting
> > > > up
> > > > > a
> > > > > > > > mirror
> > > > > > > > > > on
> > > > > > > > > > > some cloudstack/apache domain and then mirror to other
> > > > provided
> > > > > > > > > > templates.
> > > > > > > > > > >
> > > > > > > > > > > we MUST come up with a solution to deprecate '
> > > > > download.cloud.com
> > > > > > ',
> > > > > > > > > that
> > > > > > > > > > > is going to be going away any day now.
> > > > > > > > > > >
> > > > > > > > > > > i don't know the right way to solve this to be honest,
> > but
> > > if
> > > > > you
> > > > > > > > have
> > > > > > > > > > > ideas, i am willing to help.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > Lead Developer
> > > > > > > > > > >
> > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> > > > > > njohnson@ena.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > So not to re-open a can of worms, but I’m in a
> > situation
> > > > > where
> > > > > > I
> > > > > > > > need
> > > > > > > > > > > > to come up with a Marvin component test that depends
> > on a
> > > > > > > template
> > > > > > > > > > > > based on a kernel that’s relatively new, i.e., newer
> > than
> > > > > > Centos
> > > > > > > > 5.3
> > > > > > > > > /
> > > > > > > > > > > Ubuntu 10.04 .
> > > > > > > > > > > > I see openvm.eu has a suitable template (Ubuntu
> 16.0.4
> > > for
> > > > > > KVM),
> > > > > > > > but
> > > > > > > > > > > > from looking at the thread "Migrating CloudStack
> > content
> > > > from
> > > > > > > > > > > > download.cloud.com” it looks like there is
> resistance
> > to
> > > > > using
> > > > > > > > this
> > > > > > > > > at
> > > > > > > > > > > > least for hosting system vm templates over concerns
> of
> > > > > > > neutrality.
> > > > > > > > > > > > Would this be suitable for a component test?  If not,
> > > what
> > > > > is a
> > > > > > > > > > “blessed”
> > > > > > > > > > > > template location?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks in advance!
> > > > > > > > > > > >
> > > > > > > > > > > > Nathan Johnson
> > > > > > > > > > > > R&D Engineer
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > > > > Nashville, TN 37211
> > > > > > > > > > > > General Office: 615-312-6000
> > > > > > > > > > > >
> > > > > > > > > > > > website | blog | support
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > DISCLAIMER
> > > > > > > > > > ==========
> > > > > > > > > > This e-mail may contain privileged and confidential
> > > information
> > > > > > which
> > > > > > > > is
> > > > > > > > > > the property of Accelerite, a Persistent Systems
> business.
> > It
> > > > is
> > > > > > > > intended
> > > > > > > > > > only for the use of the individual or entity to which it
> is
> > > > > > > addressed.
> > > > > > > > If
> > > > > > > > > > you are not the intended recipient, you are not
> authorized
> > to
> > > > > read,
> > > > > > > > > retain,
> > > > > > > > > > copy, print, distribute or use this message. If you have
> > > > received
> > > > > > > this
> > > > > > > > > > communication in error, please notify the sender and
> delete
> > > all
> > > > > > > copies
> > > > > > > > of
> > > > > > > > > > this message. Accelerite, a Persistent Systems business
> > does
> > > > not
> > > > > > > accept
> > > > > > > > > any
> > > > > > > > > > liability for virus infected mails.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> > >
> >
>



-- 
Rafael Weingärtner

Re: Modern template hosting

Posted by Will Stevens <ws...@cloudops.com>.

@rafael: Yes, I think it makes sense to support sub-paths.  It will make
things a lot easier to work with.  The security question is an interesting
one.  I am not sure I would be the best person to answer that.  I would say
that we would want to have the MD5s hosted in the same repo as the mirror
list urls.

I do like the idea of running a web server on the management server for
hosting the system vms though.  It could be used to populate secondary
storage and could potentially give operators an easier (more efficient?)
workflow for introducing new templates.  That area could use some work...

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Mon, Feb 27, 2017 at 4:36 PM, Chiradeep Vittal <ch...@gmail.com>
wrote:

> Hashes are checked (md5 IIRC) today.
> But given the issues, I think the project should steer away from hosting
> templates except the systemvm template.
>
> On Mon, Feb 27, 2017 at 1:31 PM, Rafael Weingärtner <
> rafaelweingartner@gmail.com> wrote:
>
> > Will, I think we could support different path structures. This can
> > facilitate different deployment of mirrors based on the structure the
> host
> > has.
> >
> > Could I add something else to the discussion? Have we discussed the
> > security impacts of setting up this mirrors approach?
> > I mean, if any of the mirrors gets corrupted (let`s say by a hacker), and
> > the templates are injected with malicious code, an attacker could
> > potentially get un-monitored and unlimited access to a cloud environment.
> >
> > If we assume that the mirror may get malicious (it is not that I do not
> > trust you guys, but bad things happen), we cannot host hashes there.
> Where
> > do you think we could store Sha512 or another hash type for these
> > templates? Could we host in the newly proposed Github repo or maybe some
> > place in the ACS website?
> >
> > This would have an impact on clients (needing clear documentation) and
> our
> > code that automatically downloads System VM templates (does it check
> hashes
> > when automatically installing templates today? It may require
> > implementation changes).
> >
> > On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <ws...@cloudops.com>
> > wrote:
> >
> > > so this is what I am looking to do.  Please let me know if you have
> > > suggestions for me or think I should be solving the problem a different
> > > way.
> > >
> > > - We request a new Github repository from the ASF at:
> > > 'apache/cloudstack-mirror-list'
> > > - In this repository we track a text file in the 'gh-pages' branch
> with a
> > > list of valid download mirrors.
> > > - I build a binary to be hosted by the ASF (or at least with the ASF
> > > pointing a domain at the binary and I could potentially host it).  We
> > will
> > > see how they want to handle the hosting of the binary.
> > >
> > > The binary would expose a web server which would behave as follows:
> > > - When the 'client' requests a download url the following flow is
> kicked
> > > off:
> > > -- The mirror list is queried from github (or from a static site hosted
> > on
> > > asf, as we see fit).
> > > -- The Lat/Lon of the 'client' is determined based on their IP.
> > > -- The Lat/Lon for each of the 'mirror's is determined based on an IP
> > > lookup of the hostname.
> > > -- The closest geographical mirror is determined, the target is
> validated
> > > to be available and the user is redirected.
> > >
> > > Some questions I have right now:
> > > - Will every mirror have the same path structure to access the
> equivalent
> > > resources?
> > > - Should we support adding a path to the mirror url to specify the path
> > to
> > > the base common path?
> > > -- Example: lets say the binary is hosted on 'dl.acs.com' and there
> are
> > > three mirrors 'abc.com', 'pqr.com/files' and 'xyx.com/downloads'.
> > > -- If the path being requested is '
> > > dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would result in the
> > > following potential paths for the mirrors:
> > > -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> > > -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> > > -- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'
> > >
> > > Does this all make sense?
> > >
> > > *Will STEVENS*
> > > Lead Developer
> > >
> > > <https://goo.gl/NYZ8KK>
> > >
> > > On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <
> chiradeepv@gmail.com>
> > > wrote:
> > >
> > > > My bad. A few lines down, this has been added recently:
> > > >
> > > > this.request.setFollowRedirects(true);
> > > >
> > > > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <
> > williamstevens@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > OK. Thanks for the heads up.
> > > > >
> > > > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <ch...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Sounds workable. The downloader code in the SSVM won't follow
> > > > redirects I
> > > > > > think.
> > > > > > https://github.com/apache/cloudstack/blob/
> > > > 5511065fc20787619d9cd0444a65a3
> > > > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > > > HttpTemplateDownloader.java#L93
> > > > > > https://goo.gl/dSi0r5
> > > > > >  Might need to add
> > > > > > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > > > >
> > > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <
> > wstevens@cloudops.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > We haven't opened a ticket yet because we don't have a strategy
> > > yet.
> > > > > > >
> > > > > > > What do you guys think of this:
> > > > > > > - We setup a new github repo in the 'apache' org which consists
> > of
> > > a
> > > > > > single
> > > > > > > file with a list of active/supported mirrors.
> > > > > > > - I write a small web server, distributed as a binary, which
> can
> > be
> > > > > > hosted
> > > > > > > by ASF Infra.  This web server will query the current list of
> > > mirrors
> > > > > and
> > > > > > > will select one and then do a 302 redirect to that mirror.
> > > > > > >
> > > > > > > The act of 'choosing' a mirror could be done in a number of
> ways.
> > > > > > > - If we want to define an order, then it could just try from
> the
> > > top
> > > > of
> > > > > > the
> > > > > > > list and work its way down.  It would curl the target to make
> > sure
> > > it
> > > > > > gets
> > > > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > > > - Or, if we want to distribute the load across the mirrors, we
> > > could
> > > > > pick
> > > > > > > from the list randomly.  Again, doing a curl to verify the
> mirror
> > > is
> > > > up
> > > > > > and
> > > > > > > then doing a redirect.
> > > > > > > - If we want to get fancy, we could do a reverse IP lookup and
> > try
> > > to
> > > > > > match
> > > > > > > the requester with their closest geographical mirror.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > *Will STEVENS*
> > > > > > > Lead Developer
> > > > > > >
> > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > >
> > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > > chiradeepv@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > What steps are needed to set up a mirror? What does Infra
> need
> > to
> > > > do?
> > > > > > Has
> > > > > > > > anybody filed a ticket with Infra?
> > > > > > > >
> > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > raja.pullela@accelerite.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi will,
> > > > > > > > >
> > > > > > > > > I believe, we didn’t get to close ‘getting a mirror on
> > Apache’
> > > > > > because
> > > > > > > we
> > > > > > > > > needed someone on the Apache Infra side to close this. BTW,
> > > > > > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
> > all/most
> > > > of
> > > > > > the
> > > > > > > > > content.  Once we can close on the Apache mirror for
> hosting
> > > the
> > > > > > > > content, I
> > > > > > > > > can help assist getting the content there.
> > > > > > > > >
> > > > > > > > > For now, we have replicated the download.cloud.com content
> > to
> > > ‘
> > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > Also, we are working on a set of steps/procedure to help
> with
> > > > this
> > > > > > > > > change.  I will update everyone in about a week’s time on
> the
> > > > > > details.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Raja Pullela
> > > > > > > > > Engineering Team,
> > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > >
> > > > > > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf
> of
> > > > Will
> > > > > > > > > Stevens" <williamstevens@gmail.com on behalf of
> > > > > > wstevens@cloudops.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > unfortunately the template mirror conversation got caught
> up
> > in
> > > > > > details
> > > > > > > > and
> > > > > > > > > nobody took the lead on implementing a solution.
> > > > > > > > >
> > > > > > > > > citrix has been pinging me every couple months to say
> 'dude,
> > we
> > > > > need
> > > > > > to
> > > > > > > > > remove the dependency on download.citrix.com', but i have
> > not
> > > > had
> > > > > > the
> > > > > > > > > cycles to get in and solve the problem.  the shutdown of
> that
> > > is
> > > > > > > imminent
> > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *Will STEVENS*
> > > > > > > > > Lead Developer
> > > > > > > > >
> > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > >
> > > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > > > paul.angus@shapeblue.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Nathan,
> > > > > > > > > >
> > > > > > > > > > Ideally, if you put the template location in (or use a
> > > template
> > > > > > > defined
> > > > > > > > > > in)  test_data.py then the actual location can be
> > overridden
> > > by
> > > > > > > anyone
> > > > > > > > > > testing.
> > > > > > > > > >
> > > > > > > > > > For Trillian, we've copied all of the templates that
> people
> > > > have
> > > > > > > define
> > > > > > > > > to
> > > > > > > > > > a local repo and then replace the URLs in test_data.py to
> > > > reduce
> > > > > > > > > bandwidth
> > > > > > > > > > use and download times.
> > > > > > > > > >
> > > > > > > > > > Ie:
> > > > > > > > > >
> > > > > > > > > >             "bootableIso":
> > > > > > > > > >                 {
> > > > > > > > > >                     "displaytext": "Test Bootable ISO",
> > > > > > > > > >                     "name": "testISO",
> > > > > > > > > >                     "bootable": True,
> > > > > > > > > >                     "ispublic": False,
> > > > > > > > > >                     "url": "{{ marvin_images_location
> > > > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > > >                     "ostype": 'Other Linux (64-bit)',
> > > > > > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > > > > > >         },
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I thought that we had come up with a solution for
> > > > > > download.cloud.com
> > > > > > > ,
> > > > > > > > by
> > > > > > > > > > having a mirrorlist hosted in Community Apache 'space'
> with
> > > > > anyone
> > > > > > > able
> > > > > > > > > to
> > > > > > > > > > out themselves forward as a mirror.
> > > > > > > > > > But I must admit I lost track of whether anyone made the
> > > > > requisite
> > > > > > > > > changes
> > > > > > > > > > in code....
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Kind regards,
> > > > > > > > > >
> > > > > > > > > > Paul Angus
> > > > > > > > > >
> > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > www.shapeblue.com
> > > > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > > > > @shapeblue
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: williamstevens@gmail.com [mailto:
> > williamstevens@gmail.
> > > > com]
> > > > > > On
> > > > > > > > > > Behalf Of Will Stevens
> > > > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > > Subject: Re: Modern template hosting
> > > > > > > > > >
> > > > > > > > > > this is a hard questions.  in general, we should be
> setting
> > > up
> > > > a
> > > > > > > mirror
> > > > > > > > > on
> > > > > > > > > > some cloudstack/apache domain and then mirror to other
> > > provided
> > > > > > > > > templates.
> > > > > > > > > >
> > > > > > > > > > we MUST come up with a solution to deprecate '
> > > > download.cloud.com
> > > > > ',
> > > > > > > > that
> > > > > > > > > > is going to be going away any day now.
> > > > > > > > > >
> > > > > > > > > > i don't know the right way to solve this to be honest,
> but
> > if
> > > > you
> > > > > > > have
> > > > > > > > > > ideas, i am willing to help.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > *Will STEVENS*
> > > > > > > > > > Lead Developer
> > > > > > > > > >
> > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> > > > > njohnson@ena.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > So not to re-open a can of worms, but I’m in a
> situation
> > > > where
> > > > > I
> > > > > > > need
> > > > > > > > > > > to come up with a Marvin component test that depends
> on a
> > > > > > template
> > > > > > > > > > > based on a kernel that’s relatively new, i.e., newer
> than
> > > > > Centos
> > > > > > > 5.3
> > > > > > > > /
> > > > > > > > > > Ubuntu 10.04 .
> > > > > > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4
> > for
> > > > > KVM),
> > > > > > > but
> > > > > > > > > > > from looking at the thread "Migrating CloudStack
> content
> > > from
> > > > > > > > > > > download.cloud.com” it looks like there is resistance
> to
> > > > using
> > > > > > > this
> > > > > > > > at
> > > > > > > > > > > least for hosting system vm templates over concerns of
> > > > > > neutrality.
> > > > > > > > > > > Would this be suitable for a component test?  If not,
> > what
> > > > is a
> > > > > > > > > “blessed”
> > > > > > > > > > > template location?
> > > > > > > > > > >
> > > > > > > > > > > Thanks in advance!
> > > > > > > > > > >
> > > > > > > > > > > Nathan Johnson
> > > > > > > > > > > R&D Engineer
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > > > Nashville, TN 37211
> > > > > > > > > > > General Office: 615-312-6000
> > > > > > > > > > >
> > > > > > > > > > > website | blog | support
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > DISCLAIMER
> > > > > > > > > ==========
> > > > > > > > > This e-mail may contain privileged and confidential
> > information
> > > > > which
> > > > > > > is
> > > > > > > > > the property of Accelerite, a Persistent Systems business.
> It
> > > is
> > > > > > > intended
> > > > > > > > > only for the use of the individual or entity to which it is
> > > > > > addressed.
> > > > > > > If
> > > > > > > > > you are not the intended recipient, you are not authorized
> to
> > > > read,
> > > > > > > > retain,
> > > > > > > > > copy, print, distribute or use this message. If you have
> > > received
> > > > > > this
> > > > > > > > > communication in error, please notify the sender and delete
> > all
> > > > > > copies
> > > > > > > of
> > > > > > > > > this message. Accelerite, a Persistent Systems business
> does
> > > not
> > > > > > accept
> > > > > > > > any
> > > > > > > > > liability for virus infected mails.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Rafael Weingärtner
> >
>

Re: Modern template hosting

Posted by Chiradeep Vittal <ch...@gmail.com>.

The cloud templates provided by Ubuntu and CentOS (Debian doesn't provide
one) can work out of the box if we support http://1692.254.169.254/ as a
metadata URL. I believe this is as simple as adding this IP to the VR (+ a
DNAT rule in the VR?) for advanced zone. For Basic Zone the default router
is not the VR, so additional DNAT rule will have to be added on the xenbr0
bridge.

On Mon, Feb 27, 2017 at 5:40 PM, Pierre-Luc Dion <pd...@apache.org>
wrote:

> hi!
>
> I think we should work with distro provider to have their cloud builds work
> with cloudstack. Good example is CoreOS, it work out of the box from their
> channel builds.
> it shouldn't be too complicated to have centos, ubuntu and debian, unless
> ...
>
> For our systemvm templates, can we just change the URL for somthing like
> cloudstack.apache.org/systemvm/... ?
> and for old systemvm that are depricated but that we might want to keep,
> could we archive them into a github repo in
> https://github.com/apachecloudstack ?
>
>
>
>
> On Mon, Feb 27, 2017 at 6:28 PM, Chiradeep Vittal <ch...@gmail.com>
> wrote:
>
> > My stance is that the current workflow does a disservice to the user
> > community by letting them install / use outdated and insecure templates.
> > Now, let's assume download.cloud.com is gone forever. What do we tell
> ACS
> > users pre-4.11 as far as *built-in templates* go?
> > 1. Direct them to update templates.sql with some new URL, but with the
> same
> > dirty old templates
> > 2. Direct them to update templates.sql with some new URL, but with nice
> > templates (e.g., open.vm.eu)
> > 3. Same as (2), but document more choices.
> >
> > Now, why should things be different for 4.11 and later? Documenting the
> > steps to install templates offline is trivial (and can be scripted to a
> > large part, like cloud-install-sys-tmplt)
> >
> > For pre-4.11 users, for *systemvms*, anyway we tell them to use
> > http://cloudstack.apt-get.eu which is not controlled by ACS.
> >
> >
> > On Mon, Feb 27, 2017 at 2:50 PM, Rafael Weingärtner <
> > rafaelweingartner@gmail.com> wrote:
> >
> > > Agree with you.
> > > We need to support the current working flow. And then, define the first
> > > version that will start using the new approach.
> > >
> > > On Mon, Feb 27, 2017 at 5:36 PM, Will Stevens <ws...@cloudops.com>
> > > wrote:
> > >
> > > > I think we almost need a two pronged approach.
> > > >
> > > > 1) Get a solution in place which will enable us to document and serve
> > > > templates for legacy systems.  I will work on this.
> > > > 2) Discuss and understand how we SHOULD be handling this problem in
> the
> > > > future and in what release we can expect it.
> > > >
> > > > I think we need to do both.  I think we should start to try to really
> > > > understand what we want to deliver in (2) going forward.
> > > >
> > > > *Will STEVENS*
> > > > Lead Developer
> > > >
> > > > <https://goo.gl/NYZ8KK>
> > > >
> > > > On Mon, Feb 27, 2017 at 4:53 PM, Rafael Weingärtner <
> > > > rafaelweingartner@gmail.com> wrote:
> > > >
> > > > > My worry is exactly with system VMs templates.
> > > > >
> > > > > Currently, we indicate administrators to download them from
> > > > > http://cloudstack.apt-get.eu/systemvm/4.6/ [1]. However, the
> > > > installation
> > > > > docs do not mention the expected hashes for the file that is going
> to
> > > be
> > > > > downloaded.
> > > > > Also, I do not know the code that downloads system VMs templates
> > (when
> > > > > upgrading), but if the hash being checked is taken from the mirror
> > used
> > > > to
> > > > > download the file; the only thing it checks is that if the download
> > > > > finished successfully (no transmission errors). If we want to check
> > > > > integrity, check that the template we created is untampered; we
> need
> > to
> > > > > host and serve the hash in a secure manner.
> > > > >
> > > > > [1]
> > > > > http://docs.cloudstack.apache.org/projects/cloudstack-
> > > > installation/en/4.9/
> > > > > management-server/index.html#prepare-the-system-vm-template
> > > > >
> > > > >
> > > > > On Mon, Feb 27, 2017 at 4:36 PM, Chiradeep Vittal <
> > > chiradeepv@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hashes are checked (md5 IIRC) today.
> > > > > > But given the issues, I think the project should steer away from
> > > > hosting
> > > > > > templates except the systemvm template.
> > > > > >
> > > > > > On Mon, Feb 27, 2017 at 1:31 PM, Rafael Weingärtner <
> > > > > > rafaelweingartner@gmail.com> wrote:
> > > > > >
> > > > > > > Will, I think we could support different path structures. This
> > can
> > > > > > > facilitate different deployment of mirrors based on the
> structure
> > > the
> > > > > > host
> > > > > > > has.
> > > > > > >
> > > > > > > Could I add something else to the discussion? Have we discussed
> > the
> > > > > > > security impacts of setting up this mirrors approach?
> > > > > > > I mean, if any of the mirrors gets corrupted (let`s say by a
> > > hacker),
> > > > > and
> > > > > > > the templates are injected with malicious code, an attacker
> could
> > > > > > > potentially get un-monitored and unlimited access to a cloud
> > > > > environment.
> > > > > > >
> > > > > > > If we assume that the mirror may get malicious (it is not that
> I
> > do
> > > > not
> > > > > > > trust you guys, but bad things happen), we cannot host hashes
> > > there.
> > > > > > Where
> > > > > > > do you think we could store Sha512 or another hash type for
> these
> > > > > > > templates? Could we host in the newly proposed Github repo or
> > maybe
> > > > > some
> > > > > > > place in the ACS website?
> > > > > > >
> > > > > > > This would have an impact on clients (needing clear
> > documentation)
> > > > and
> > > > > > our
> > > > > > > code that automatically downloads System VM templates (does it
> > > check
> > > > > > hashes
> > > > > > > when automatically installing templates today? It may require
> > > > > > > implementation changes).
> > > > > > >
> > > > > > > On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <
> > > wstevens@cloudops.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > so this is what I am looking to do.  Please let me know if
> you
> > > have
> > > > > > > > suggestions for me or think I should be solving the problem a
> > > > > different
> > > > > > > > way.
> > > > > > > >
> > > > > > > > - We request a new Github repository from the ASF at:
> > > > > > > > 'apache/cloudstack-mirror-list'
> > > > > > > > - In this repository we track a text file in the 'gh-pages'
> > > branch
> > > > > > with a
> > > > > > > > list of valid download mirrors.
> > > > > > > > - I build a binary to be hosted by the ASF (or at least with
> > the
> > > > ASF
> > > > > > > > pointing a domain at the binary and I could potentially host
> > it).
> > > > We
> > > > > > > will
> > > > > > > > see how they want to handle the hosting of the binary.
> > > > > > > >
> > > > > > > > The binary would expose a web server which would behave as
> > > follows:
> > > > > > > > - When the 'client' requests a download url the following
> flow
> > is
> > > > > > kicked
> > > > > > > > off:
> > > > > > > > -- The mirror list is queried from github (or from a static
> > site
> > > > > hosted
> > > > > > > on
> > > > > > > > asf, as we see fit).
> > > > > > > > -- The Lat/Lon of the 'client' is determined based on their
> IP.
> > > > > > > > -- The Lat/Lon for each of the 'mirror's is determined based
> on
> > > an
> > > > IP
> > > > > > > > lookup of the hostname.
> > > > > > > > -- The closest geographical mirror is determined, the target
> is
> > > > > > validated
> > > > > > > > to be available and the user is redirected.
> > > > > > > >
> > > > > > > > Some questions I have right now:
> > > > > > > > - Will every mirror have the same path structure to access
> the
> > > > > > equivalent
> > > > > > > > resources?
> > > > > > > > - Should we support adding a path to the mirror url to
> specify
> > > the
> > > > > path
> > > > > > > to
> > > > > > > > the base common path?
> > > > > > > > -- Example: lets say the binary is hosted on 'dl.acs.com'
> and
> > > > there
> > > > > > are
> > > > > > > > three mirrors 'abc.com', 'pqr.com/files' and '
> > xyx.com/downloads
> > > '.
> > > > > > > > -- If the path being requested is '
> > > > > > > > dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would
> > result
> > > in
> > > > > the
> > > > > > > > following potential paths for the mirrors:
> > > > > > > > -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > > > -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > > > -- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > > >
> > > > > > > > Does this all make sense?
> > > > > > > >
> > > > > > > > *Will STEVENS*
> > > > > > > > Lead Developer
> > > > > > > >
> > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > >
> > > > > > > > On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <
> > > > > > chiradeepv@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > My bad. A few lines down, this has been added recently:
> > > > > > > > >
> > > > > > > > > this.request.setFollowRedirects(true);
> > > > > > > > >
> > > > > > > > > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <
> > > > > > > williamstevens@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > OK. Thanks for the heads up.
> > > > > > > > > >
> > > > > > > > > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > > chiradeepv@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Sounds workable. The downloader code in the SSVM won't
> > > follow
> > > > > > > > > redirects I
> > > > > > > > > > > think.
> > > > > > > > > > > https://github.com/apache/cloudstack/blob/
> > > > > > > > > 5511065fc20787619d9cd0444a65a3
> > > > > > > > > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > > > > > > > > HttpTemplateDownloader.java#L93
> > > > > > > > > > > https://goo.gl/dSi0r5
> > > > > > > > > > >  Might need to add
> > > > > > > > > > > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <
> > > > > > > wstevens@cloudops.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > We haven't opened a ticket yet because we don't have
> a
> > > > > strategy
> > > > > > > > yet.
> > > > > > > > > > > >
> > > > > > > > > > > > What do you guys think of this:
> > > > > > > > > > > > - We setup a new github repo in the 'apache' org
> which
> > > > > consists
> > > > > > > of
> > > > > > > > a
> > > > > > > > > > > single
> > > > > > > > > > > > file with a list of active/supported mirrors.
> > > > > > > > > > > > - I write a small web server, distributed as a
> binary,
> > > > which
> > > > > > can
> > > > > > > be
> > > > > > > > > > > hosted
> > > > > > > > > > > > by ASF Infra.  This web server will query the current
> > > list
> > > > of
> > > > > > > > mirrors
> > > > > > > > > > and
> > > > > > > > > > > > will select one and then do a 302 redirect to that
> > > mirror.
> > > > > > > > > > > >
> > > > > > > > > > > > The act of 'choosing' a mirror could be done in a
> > number
> > > of
> > > > > > ways.
> > > > > > > > > > > > - If we want to define an order, then it could just
> try
> > > > from
> > > > > > the
> > > > > > > > top
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > list and work its way down.  It would curl the target
> > to
> > > > make
> > > > > > > sure
> > > > > > > > it
> > > > > > > > > > > gets
> > > > > > > > > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > > > > > > > > - Or, if we want to distribute the load across the
> > > mirrors,
> > > > > we
> > > > > > > > could
> > > > > > > > > > pick
> > > > > > > > > > > > from the list randomly.  Again, doing a curl to
> verify
> > > the
> > > > > > mirror
> > > > > > > > is
> > > > > > > > > up
> > > > > > > > > > > and
> > > > > > > > > > > > then doing a redirect.
> > > > > > > > > > > > - If we want to get fancy, we could do a reverse IP
> > > lookup
> > > > > and
> > > > > > > try
> > > > > > > > to
> > > > > > > > > > > match
> > > > > > > > > > > > the requester with their closest geographical mirror.
> > > > > > > > > > > >
> > > > > > > > > > > > Thoughts?
> > > > > > > > > > > >
> > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > Lead Developer
> > > > > > > > > > > >
> > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > > > > > > > chiradeepv@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > What steps are needed to set up a mirror? What does
> > > Infra
> > > > > > need
> > > > > > > to
> > > > > > > > > do?
> > > > > > > > > > > Has
> > > > > > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > > > > > > raja.pullela@accelerite.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi will,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I believe, we didn’t get to close ‘getting a
> mirror
> > > on
> > > > > > > Apache’
> > > > > > > > > > > because
> > > > > > > > > > > > we
> > > > > > > > > > > > > > needed someone on the Apache Infra side to close
> > > this.
> > > > > BTW,
> > > > > > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages
> this?)
> > > has
> > > > > > > all/most
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > > content.  Once we can close on the Apache mirror
> > for
> > > > > > hosting
> > > > > > > > the
> > > > > > > > > > > > > content, I
> > > > > > > > > > > > > > can help assist getting the content there.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For now, we have replicated the
> download.cloud.com
> > > > > content
> > > > > > > to
> > > > > > > > ‘
> > > > > > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > > > > > Also, we are working on a set of steps/procedure
> to
> > > > help
> > > > > > with
> > > > > > > > > this
> > > > > > > > > > > > > > change.  I will update everyone in about a week’s
> > > time
> > > > on
> > > > > > the
> > > > > > > > > > > details.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Raja Pullela
> > > > > > > > > > > > > > Engineering Team,
> > > > > > > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com
> on
> > > > > behalf
> > > > > > of
> > > > > > > > > Will
> > > > > > > > > > > > > > Stevens" <williamstevens@gmail.com on behalf of
> > > > > > > > > > > wstevens@cloudops.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > unfortunately the template mirror conversation
> got
> > > > caught
> > > > > > up
> > > > > > > in
> > > > > > > > > > > details
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > nobody took the lead on implementing a solution.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > citrix has been pinging me every couple months to
> > say
> > > > > > 'dude,
> > > > > > > we
> > > > > > > > > > need
> > > > > > > > > > > to
> > > > > > > > > > > > > > remove the dependency on download.citrix.com',
> > but i
> > > > > have
> > > > > > > not
> > > > > > > > > had
> > > > > > > > > > > the
> > > > > > > > > > > > > > cycles to get in and solve the problem.  the
> > shutdown
> > > > of
> > > > > > that
> > > > > > > > is
> > > > > > > > > > > > imminent
> > > > > > > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Nathan,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Ideally, if you put the template location in
> (or
> > > use
> > > > a
> > > > > > > > template
> > > > > > > > > > > > defined
> > > > > > > > > > > > > > > in)  test_data.py then the actual location can
> be
> > > > > > > overridden
> > > > > > > > by
> > > > > > > > > > > > anyone
> > > > > > > > > > > > > > > testing.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For Trillian, we've copied all of the templates
> > > that
> > > > > > people
> > > > > > > > > have
> > > > > > > > > > > > define
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > a local repo and then replace the URLs in
> > > > test_data.py
> > > > > to
> > > > > > > > > reduce
> > > > > > > > > > > > > > bandwidth
> > > > > > > > > > > > > > > use and download times.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Ie:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >             "bootableIso":
> > > > > > > > > > > > > > >                 {
> > > > > > > > > > > > > > >                     "displaytext": "Test
> Bootable
> > > > ISO",
> > > > > > > > > > > > > > >                     "name": "testISO",
> > > > > > > > > > > > > > >                     "bootable": True,
> > > > > > > > > > > > > > >                     "ispublic": False,
> > > > > > > > > > > > > > >                     "url": "{{
> > > marvin_images_location
> > > > > > > > > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > > > > > > > >                     "ostype": 'Other Linux
> > > (64-bit)',
> > > > > > > > > > > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > > > > > > > > > > >         },
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I thought that we had come up with a solution
> for
> > > > > > > > > > > download.cloud.com
> > > > > > > > > > > > ,
> > > > > > > > > > > > > by
> > > > > > > > > > > > > > > having a mirrorlist hosted in Community Apache
> > > > 'space'
> > > > > > with
> > > > > > > > > > anyone
> > > > > > > > > > > > able
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > out themselves forward as a mirror.
> > > > > > > > > > > > > > > But I must admit I lost track of whether anyone
> > > made
> > > > > the
> > > > > > > > > > requisite
> > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > in code....
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Kind regards,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Paul Angus
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > > > > > www.shapeblue.com
> > > > > > > > > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N
> > 4HSUK
> > > > > > > > > > > > > > > @shapeblue
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > > > > From: williamstevens@gmail.com [mailto:
> > > > > > > williamstevens@gmail.
> > > > > > > > > com]
> > > > > > > > > > > On
> > > > > > > > > > > > > > > Behalf Of Will Stevens
> > > > > > > > > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > > > > > > > Subject: Re: Modern template hosting
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > this is a hard questions.  in general, we
> should
> > be
> > > > > > setting
> > > > > > > > up
> > > > > > > > > a
> > > > > > > > > > > > mirror
> > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > some cloudstack/apache domain and then mirror
> to
> > > > other
> > > > > > > > provided
> > > > > > > > > > > > > > templates.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > we MUST come up with a solution to deprecate '
> > > > > > > > > download.cloud.com
> > > > > > > > > > ',
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > is going to be going away any day now.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > i don't know the right way to solve this to be
> > > > honest,
> > > > > > but
> > > > > > > if
> > > > > > > > > you
> > > > > > > > > > > > have
> > > > > > > > > > > > > > > ideas, i am willing to help.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan
> Johnson
> > <
> > > > > > > > > > njohnson@ena.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > So not to re-open a can of worms, but I’m in
> a
> > > > > > situation
> > > > > > > > > where
> > > > > > > > > > I
> > > > > > > > > > > > need
> > > > > > > > > > > > > > > > to come up with a Marvin component test that
> > > > depends
> > > > > > on a
> > > > > > > > > > > template
> > > > > > > > > > > > > > > > based on a kernel that’s relatively new,
> i.e.,
> > > > newer
> > > > > > than
> > > > > > > > > > Centos
> > > > > > > > > > > > 5.3
> > > > > > > > > > > > > /
> > > > > > > > > > > > > > > Ubuntu 10.04 .
> > > > > > > > > > > > > > > > I see openvm.eu has a suitable template
> > (Ubuntu
> > > > > 16.0.4
> > > > > > > for
> > > > > > > > > > KVM),
> > > > > > > > > > > > but
> > > > > > > > > > > > > > > > from looking at the thread "Migrating
> > CloudStack
> > > > > > content
> > > > > > > > from
> > > > > > > > > > > > > > > > download.cloud.com” it looks like there is
> > > > > resistance
> > > > > > to
> > > > > > > > > using
> > > > > > > > > > > > this
> > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > least for hosting system vm templates over
> > > concerns
> > > > > of
> > > > > > > > > > > neutrality.
> > > > > > > > > > > > > > > > Would this be suitable for a component test?
> > If
> > > > not,
> > > > > > > what
> > > > > > > > > is a
> > > > > > > > > > > > > > “blessed”
> > > > > > > > > > > > > > > > template location?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks in advance!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Nathan Johnson
> > > > > > > > > > > > > > > > R&D Engineer
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > > > > > > > > Nashville, TN 37211
> > > > > > > > > > > > > > > > General Office: 615-312-6000
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > website | blog | support
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > DISCLAIMER
> > > > > > > > > > > > > > ==========
> > > > > > > > > > > > > > This e-mail may contain privileged and
> confidential
> > > > > > > information
> > > > > > > > > > which
> > > > > > > > > > > > is
> > > > > > > > > > > > > > the property of Accelerite, a Persistent Systems
> > > > > business.
> > > > > > It
> > > > > > > > is
> > > > > > > > > > > > intended
> > > > > > > > > > > > > > only for the use of the individual or entity to
> > which
> > > > it
> > > > > is
> > > > > > > > > > > addressed.
> > > > > > > > > > > > If
> > > > > > > > > > > > > > you are not the intended recipient, you are not
> > > > > authorized
> > > > > > to
> > > > > > > > > read,
> > > > > > > > > > > > > retain,
> > > > > > > > > > > > > > copy, print, distribute or use this message. If
> you
> > > > have
> > > > > > > > received
> > > > > > > > > > > this
> > > > > > > > > > > > > > communication in error, please notify the sender
> > and
> > > > > delete
> > > > > > > all
> > > > > > > > > > > copies
> > > > > > > > > > > > of
> > > > > > > > > > > > > > this message. Accelerite, a Persistent Systems
> > > business
> > > > > > does
> > > > > > > > not
> > > > > > > > > > > accept
> > > > > > > > > > > > > any
> > > > > > > > > > > > > > liability for virus infected mails.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Rafael Weingärtner
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rafael Weingärtner
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> > >
> >
>

Re: Modern template hosting

Posted by Wido den Hollander <wi...@widodh.nl>.

> Op 28 februari 2017 om 19:08 schreef Chiradeep Vittal <ch...@gmail.com>:
> 
> 
> Yes, that's my preferred solution. But there's a couple of scenarios to be
> dealt with:
>  1. standard isolated networks where the VR is the default router
>  2. basic zone where the VR is not the default router
>  3. isolated networks where the VR is not the default router
> 
> For (2) and (3) you may have to send down a static route in the DHCP
> response to point 169.254.169.254 to the VR (or do it in all cases,
> shouldn't be a big deal).

I didn't know we could do that with DHCP. Looks cool! If we could do that we can use those templates easily and don't have to worry about our own ones.

Wido

> 
> 
> On Tue, Feb 28, 2017 at 9:57 AM, Wido den Hollander <wi...@widodh.nl> wrote:
> 
> >
> > > Op 28 februari 2017 om 18:55 schreef Chiradeep Vittal <
> > chiradeepv@gmail.com>:
> > >
> > >
> > > To repeat what was discussed earlier in the thread: the cloud templates
> > > provided by Ubuntu and CentOS do not work because they expect the
> > metadata
> > > URL to be at http://169.254.269.254/. Well, they work, but they take
> > > awfully long to time out and after they time out, you still wouldn't  be
> > > able to login since they do not install the ssh keys.
> >
> > But that is a 'simple fix' in the VR. Like you mentioned, add a DNAT rule
> > in the VR and have the VR handle the traffic.
> >
> > Wido
> >
> > >
> > > On Tue, Feb 28, 2017 at 9:34 AM, Wido den Hollander <wi...@widodh.nl>
> > wrote:
> > >
> > > >
> > > > > Op 28 februari 2017 om 2:40 schreef Pierre-Luc Dion <
> > pdion891@apache.org
> > > > >:
> > > > >
> > > > >
> > > > > hi!
> > > > >
> > > > > I think we should work with distro provider to have their cloud
> > builds
> > > > work
> > > > > with cloudstack. Good example is CoreOS, it work out of the box from
> > > > their
> > > > > channel builds.
> > > > > it shouldn't be too complicated to have centos, ubuntu and debian,
> > unless
> > > > > ...
> > > >
> > > > I fully agree. I think that we shouldn't bother about VM templates.
> > CentOS
> > > > and Ubuntu both have a proper template which we can include.
> > > >
> > > > The less we host as a project, the better.
> > > >
> > > > Keep in mind that we host a Open Source project. the RPM and DEB
> > packages
> > > > on cloudstack.apt-get.eu are a service, but we release source as a
> > > > project.
> > > >
> > > > Wido
> > > >
> > > > >
> > > > > For our systemvm templates, can we just change the URL for somthing
> > like
> > > > > cloudstack.apache.org/systemvm/... ?
> > > > > and for old systemvm that are depricated but that we might want to
> > keep,
> > > > > could we archive them into a github repo in
> > > > > https://github.com/apachecloudstack ?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Feb 27, 2017 at 6:28 PM, Chiradeep Vittal <
> > chiradeepv@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > My stance is that the current workflow does a disservice to the
> > user
> > > > > > community by letting them install / use outdated and insecure
> > > > templates.
> > > > > > Now, let's assume download.cloud.com is gone forever. What do we
> > tell
> > > > ACS
> > > > > > users pre-4.11 as far as *built-in templates* go?
> > > > > > 1. Direct them to update templates.sql with some new URL, but with
> > the
> > > > same
> > > > > > dirty old templates
> > > > > > 2. Direct them to update templates.sql with some new URL, but with
> > nice
> > > > > > templates (e.g., open.vm.eu)
> > > > > > 3. Same as (2), but document more choices.
> > > > > >
> > > > > > Now, why should things be different for 4.11 and later?
> > Documenting the
> > > > > > steps to install templates offline is trivial (and can be scripted
> > to a
> > > > > > large part, like cloud-install-sys-tmplt)
> > > > > >
> > > > > > For pre-4.11 users, for *systemvms*, anyway we tell them to use
> > > > > > http://cloudstack.apt-get.eu which is not controlled by ACS.
> > > > > >
> > > > > >
> > > > > > On Mon, Feb 27, 2017 at 2:50 PM, Rafael Weingärtner <
> > > > > > rafaelweingartner@gmail.com> wrote:
> > > > > >
> > > > > > > Agree with you.
> > > > > > > We need to support the current working flow. And then, define the
> > > > first
> > > > > > > version that will start using the new approach.
> > > > > > >
> > > > > > > On Mon, Feb 27, 2017 at 5:36 PM, Will Stevens <
> > wstevens@cloudops.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I think we almost need a two pronged approach.
> > > > > > > >
> > > > > > > > 1) Get a solution in place which will enable us to document and
> > > > serve
> > > > > > > > templates for legacy systems.  I will work on this.
> > > > > > > > 2) Discuss and understand how we SHOULD be handling this
> > problem
> > > > in the
> > > > > > > > future and in what release we can expect it.
> > > > > > > >
> > > > > > > > I think we need to do both.  I think we should start to try to
> > > > really
> > > > > > > > understand what we want to deliver in (2) going forward.
> > > > > > > >
> > > > > > > > *Will STEVENS*
> > > > > > > > Lead Developer
> > > > > > > >
> > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > >
> > > > > > > > On Mon, Feb 27, 2017 at 4:53 PM, Rafael Weingärtner <
> > > > > > > > rafaelweingartner@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > My worry is exactly with system VMs templates.
> > > > > > > > >
> > > > > > > > > Currently, we indicate administrators to download them from
> > > > > > > > > http://cloudstack.apt-get.eu/systemvm/4.6/ [1]. However, the
> > > > > > > > installation
> > > > > > > > > docs do not mention the expected hashes for the file that is
> > > > going to
> > > > > > > be
> > > > > > > > > downloaded.
> > > > > > > > > Also, I do not know the code that downloads system VMs
> > templates
> > > > > > (when
> > > > > > > > > upgrading), but if the hash being checked is taken from the
> > > > mirror
> > > > > > used
> > > > > > > > to
> > > > > > > > > download the file; the only thing it checks is that if the
> > > > download
> > > > > > > > > finished successfully (no transmission errors). If we want to
> > > > check
> > > > > > > > > integrity, check that the template we created is untampered;
> > we
> > > > need
> > > > > > to
> > > > > > > > > host and serve the hash in a secure manner.
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > > http://docs.cloudstack.apache.org/projects/cloudstack-
> > > > > > > > installation/en/4.9/
> > > > > > > > > management-server/index.html#prepare-the-system-vm-template
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Feb 27, 2017 at 4:36 PM, Chiradeep Vittal <
> > > > > > > chiradeepv@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hashes are checked (md5 IIRC) today.
> > > > > > > > > > But given the issues, I think the project should steer away
> > > > from
> > > > > > > > hosting
> > > > > > > > > > templates except the systemvm template.
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 27, 2017 at 1:31 PM, Rafael Weingärtner <
> > > > > > > > > > rafaelweingartner@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Will, I think we could support different path structures.
> > > > This
> > > > > > can
> > > > > > > > > > > facilitate different deployment of mirrors based on the
> > > > structure
> > > > > > > the
> > > > > > > > > > host
> > > > > > > > > > > has.
> > > > > > > > > > >
> > > > > > > > > > > Could I add something else to the discussion? Have we
> > > > discussed
> > > > > > the
> > > > > > > > > > > security impacts of setting up this mirrors approach?
> > > > > > > > > > > I mean, if any of the mirrors gets corrupted (let`s say
> > by a
> > > > > > > hacker),
> > > > > > > > > and
> > > > > > > > > > > the templates are injected with malicious code, an
> > attacker
> > > > could
> > > > > > > > > > > potentially get un-monitored and unlimited access to a
> > cloud
> > > > > > > > > environment.
> > > > > > > > > > >
> > > > > > > > > > > If we assume that the mirror may get malicious (it is not
> > > > that I
> > > > > > do
> > > > > > > > not
> > > > > > > > > > > trust you guys, but bad things happen), we cannot host
> > hashes
> > > > > > > there.
> > > > > > > > > > Where
> > > > > > > > > > > do you think we could store Sha512 or another hash type
> > for
> > > > these
> > > > > > > > > > > templates? Could we host in the newly proposed Github
> > repo or
> > > > > > maybe
> > > > > > > > > some
> > > > > > > > > > > place in the ACS website?
> > > > > > > > > > >
> > > > > > > > > > > This would have an impact on clients (needing clear
> > > > > > documentation)
> > > > > > > > and
> > > > > > > > > > our
> > > > > > > > > > > code that automatically downloads System VM templates
> > (does
> > > > it
> > > > > > > check
> > > > > > > > > > hashes
> > > > > > > > > > > when automatically installing templates today? It may
> > require
> > > > > > > > > > > implementation changes).
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <
> > > > > > > wstevens@cloudops.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > so this is what I am looking to do.  Please let me
> > know if
> > > > you
> > > > > > > have
> > > > > > > > > > > > suggestions for me or think I should be solving the
> > > > problem a
> > > > > > > > > different
> > > > > > > > > > > > way.
> > > > > > > > > > > >
> > > > > > > > > > > > - We request a new Github repository from the ASF at:
> > > > > > > > > > > > 'apache/cloudstack-mirror-list'
> > > > > > > > > > > > - In this repository we track a text file in the
> > 'gh-pages'
> > > > > > > branch
> > > > > > > > > > with a
> > > > > > > > > > > > list of valid download mirrors.
> > > > > > > > > > > > - I build a binary to be hosted by the ASF (or at least
> > > > with
> > > > > > the
> > > > > > > > ASF
> > > > > > > > > > > > pointing a domain at the binary and I could potentially
> > > > host
> > > > > > it).
> > > > > > > > We
> > > > > > > > > > > will
> > > > > > > > > > > > see how they want to handle the hosting of the binary.
> > > > > > > > > > > >
> > > > > > > > > > > > The binary would expose a web server which would
> > behave as
> > > > > > > follows:
> > > > > > > > > > > > - When the 'client' requests a download url the
> > following
> > > > flow
> > > > > > is
> > > > > > > > > > kicked
> > > > > > > > > > > > off:
> > > > > > > > > > > > -- The mirror list is queried from github (or from a
> > static
> > > > > > site
> > > > > > > > > hosted
> > > > > > > > > > > on
> > > > > > > > > > > > asf, as we see fit).
> > > > > > > > > > > > -- The Lat/Lon of the 'client' is determined based on
> > > > their IP.
> > > > > > > > > > > > -- The Lat/Lon for each of the 'mirror's is determined
> > > > based on
> > > > > > > an
> > > > > > > > IP
> > > > > > > > > > > > lookup of the hostname.
> > > > > > > > > > > > -- The closest geographical mirror is determined, the
> > > > target is
> > > > > > > > > > validated
> > > > > > > > > > > > to be available and the user is redirected.
> > > > > > > > > > > >
> > > > > > > > > > > > Some questions I have right now:
> > > > > > > > > > > > - Will every mirror have the same path structure to
> > access
> > > > the
> > > > > > > > > > equivalent
> > > > > > > > > > > > resources?
> > > > > > > > > > > > - Should we support adding a path to the mirror url to
> > > > specify
> > > > > > > the
> > > > > > > > > path
> > > > > > > > > > > to
> > > > > > > > > > > > the base common path?
> > > > > > > > > > > > -- Example: lets say the binary is hosted on '
> > dl.acs.com'
> > > > and
> > > > > > > > there
> > > > > > > > > > are
> > > > > > > > > > > > three mirrors 'abc.com', 'pqr.com/files' and '
> > > > > > xyx.com/downloads
> > > > > > > '.
> > > > > > > > > > > > -- If the path being requested is '
> > > > > > > > > > > > dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it
> > would
> > > > > > result
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > following potential paths for the mirrors:
> > > > > > > > > > > > -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > > > > > > > -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > > > > > > > -- 'xyz.com/downloads/templates/
> > systemvm-4.6.xen.vhd.bz2'
> > > > > > > > > > > >
> > > > > > > > > > > > Does this all make sense?
> > > > > > > > > > > >
> > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > Lead Developer
> > > > > > > > > > > >
> > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <
> > > > > > > > > > chiradeepv@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > My bad. A few lines down, this has been added
> > recently:
> > > > > > > > > > > > >
> > > > > > > > > > > > > this.request.setFollowRedirects(true);
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <
> > > > > > > > > > > williamstevens@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > OK. Thanks for the heads up.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > > > > > > chiradeepv@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Sounds workable. The downloader code in the SSVM
> > > > won't
> > > > > > > follow
> > > > > > > > > > > > > redirects I
> > > > > > > > > > > > > > > think.
> > > > > > > > > > > > > > > https://github.com/apache/cloudstack/blob/
> > > > > > > > > > > > > 5511065fc20787619d9cd0444a65a3
> > > > > > > > > > > > > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > > > > > > > > > > > > HttpTemplateDownloader.java#L93
> > > > > > > > > > > > > > > https://goo.gl/dSi0r5
> > > > > > > > > > > > > > >  Might need to add
> > > > > > > > > > > > > > > client.setRedirectStrategy(new
> > > > LaxRedirectStrategy());
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <
> > > > > > > > > > > wstevens@cloudops.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > We haven't opened a ticket yet because we don't
> > > > have a
> > > > > > > > > strategy
> > > > > > > > > > > > yet.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > What do you guys think of this:
> > > > > > > > > > > > > > > > - We setup a new github repo in the 'apache'
> > org
> > > > which
> > > > > > > > > consists
> > > > > > > > > > > of
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > file with a list of active/supported mirrors.
> > > > > > > > > > > > > > > > - I write a small web server, distributed as a
> > > > binary,
> > > > > > > > which
> > > > > > > > > > can
> > > > > > > > > > > be
> > > > > > > > > > > > > > > hosted
> > > > > > > > > > > > > > > > by ASF Infra.  This web server will query the
> > > > current
> > > > > > > list
> > > > > > > > of
> > > > > > > > > > > > mirrors
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > will select one and then do a 302 redirect to
> > that
> > > > > > > mirror.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The act of 'choosing' a mirror could be done
> > in a
> > > > > > number
> > > > > > > of
> > > > > > > > > > ways.
> > > > > > > > > > > > > > > > - If we want to define an order, then it could
> > > > just try
> > > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > > > top
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > list and work its way down.  It would curl the
> > > > target
> > > > > > to
> > > > > > > > make
> > > > > > > > > > > sure
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > a 200 and if it does, it would do a 302
> > redirect.
> > > > > > > > > > > > > > > > - Or, if we want to distribute the load across
> > the
> > > > > > > mirrors,
> > > > > > > > > we
> > > > > > > > > > > > could
> > > > > > > > > > > > > > pick
> > > > > > > > > > > > > > > > from the list randomly.  Again, doing a curl to
> > > > verify
> > > > > > > the
> > > > > > > > > > mirror
> > > > > > > > > > > > is
> > > > > > > > > > > > > up
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > then doing a redirect.
> > > > > > > > > > > > > > > > - If we want to get fancy, we could do a
> > reverse IP
> > > > > > > lookup
> > > > > > > > > and
> > > > > > > > > > > try
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > match
> > > > > > > > > > > > > > > > the requester with their closest geographical
> > > > mirror.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thoughts?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep
> > Vittal
> > > > <
> > > > > > > > > > > > > > chiradeepv@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > What steps are needed to set up a mirror?
> > What
> > > > does
> > > > > > > Infra
> > > > > > > > > > need
> > > > > > > > > > > to
> > > > > > > > > > > > > do?
> > > > > > > > > > > > > > > Has
> > > > > > > > > > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja
> > Pullela <
> > > > > > > > > > > > > > > > > raja.pullela@accelerite.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi will,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I believe, we didn’t get to close ‘getting
> > a
> > > > mirror
> > > > > > > on
> > > > > > > > > > > Apache’
> > > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > needed someone on the Apache Infra side to
> > > > close
> > > > > > > this.
> > > > > > > > > BTW,
> > > > > > > > > > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages
> > > > this?)
> > > > > > > has
> > > > > > > > > > > all/most
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > content.  Once we can close on the Apache
> > > > mirror
> > > > > > for
> > > > > > > > > > hosting
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > content, I
> > > > > > > > > > > > > > > > > > can help assist getting the content there.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > For now, we have replicated the
> > > > download.cloud.com
> > > > > > > > > content
> > > > > > > > > > > to
> > > > > > > > > > > > ‘
> > > > > > > > > > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > > > > > > > > > Also, we are working on a set of
> > > > steps/procedure to
> > > > > > > > help
> > > > > > > > > > with
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > change.  I will update everyone in about a
> > > > week’s
> > > > > > > time
> > > > > > > > on
> > > > > > > > > > the
> > > > > > > > > > > > > > > details.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > Raja Pullela
> > > > > > > > > > > > > > > > > > Engineering Team,
> > > > > > > > > > > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > > > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On 2/24/17, 11:23 PM, "
> > > > williamstevens@gmail.com on
> > > > > > > > > behalf
> > > > > > > > > > of
> > > > > > > > > > > > > Will
> > > > > > > > > > > > > > > > > > Stevens" <williamstevens@gmail.com on
> > behalf
> > > > of
> > > > > > > > > > > > > > > wstevens@cloudops.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > unfortunately the template mirror
> > conversation
> > > > got
> > > > > > > > caught
> > > > > > > > > > up
> > > > > > > > > > > in
> > > > > > > > > > > > > > > details
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > nobody took the lead on implementing a
> > > > solution.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > citrix has been pinging me every couple
> > months
> > > > to
> > > > > > say
> > > > > > > > > > 'dude,
> > > > > > > > > > > we
> > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > remove the dependency on
> > download.citrix.com',
> > > > > > but i
> > > > > > > > > have
> > > > > > > > > > > not
> > > > > > > > > > > > > had
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > cycles to get in and solve the problem.
> > the
> > > > > > shutdown
> > > > > > > > of
> > > > > > > > > > that
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > imminent
> > > > > > > > > > > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul
> > Angus <
> > > > > > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Nathan,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Ideally, if you put the template
> > location in
> > > > (or
> > > > > > > use
> > > > > > > > a
> > > > > > > > > > > > template
> > > > > > > > > > > > > > > > defined
> > > > > > > > > > > > > > > > > > > in)  test_data.py then the actual
> > location
> > > > can be
> > > > > > > > > > > overridden
> > > > > > > > > > > > by
> > > > > > > > > > > > > > > > anyone
> > > > > > > > > > > > > > > > > > > testing.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > For Trillian, we've copied all of the
> > > > templates
> > > > > > > that
> > > > > > > > > > people
> > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > define
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > a local repo and then replace the URLs in
> > > > > > > > test_data.py
> > > > > > > > > to
> > > > > > > > > > > > > reduce
> > > > > > > > > > > > > > > > > > bandwidth
> > > > > > > > > > > > > > > > > > > use and download times.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Ie:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >             "bootableIso":
> > > > > > > > > > > > > > > > > > >                 {
> > > > > > > > > > > > > > > > > > >                     "displaytext": "Test
> > > > Bootable
> > > > > > > > ISO",
> > > > > > > > > > > > > > > > > > >                     "name": "testISO",
> > > > > > > > > > > > > > > > > > >                     "bootable": True,
> > > > > > > > > > > > > > > > > > >                     "ispublic": False,
> > > > > > > > > > > > > > > > > > >                     "url": "{{
> > > > > > > marvin_images_location
> > > > > > > > > > > > > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > > > > > > > > > > > >                     "ostype": 'Other
> > Linux
> > > > > > > (64-bit)',
> > > > > > > > > > > > > > > > > > >                     "mode":
> > 'HTTP_DOWNLOAD'
> > > > > > > > > > > > > > > > > > >         },
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I thought that we had come up with a
> > > > solution for
> > > > > > > > > > > > > > > download.cloud.com
> > > > > > > > > > > > > > > > ,
> > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > having a mirrorlist hosted in Community
> > > > Apache
> > > > > > > > 'space'
> > > > > > > > > > with
> > > > > > > > > > > > > > anyone
> > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > out themselves forward as a mirror.
> > > > > > > > > > > > > > > > > > > But I must admit I lost track of whether
> > > > anyone
> > > > > > > made
> > > > > > > > > the
> > > > > > > > > > > > > > requisite
> > > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > > in code....
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Kind regards,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Paul Angus
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > > > > > > > > > www.shapeblue.com
> > > > > > > > > > > > > > > > > > > 53 Chandos Place, Covent Garden, London
> > WC2N
> > > > > > 4HSUK
> > > > > > > > > > > > > > > > > > > @shapeblue
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > > > > > > > > From: williamstevens@gmail.com [mailto:
> > > > > > > > > > > williamstevens@gmail.
> > > > > > > > > > > > > com]
> > > > > > > > > > > > > > > On
> > > > > > > > > > > > > > > > > > > Behalf Of Will Stevens
> > > > > > > > > > > > > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > > > > > > > > > > > Subject: Re: Modern template hosting
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > this is a hard questions.  in general, we
> > > > should
> > > > > > be
> > > > > > > > > > setting
> > > > > > > > > > > > up
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > mirror
> > > > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > some cloudstack/apache domain and then
> > > > mirror to
> > > > > > > > other
> > > > > > > > > > > > provided
> > > > > > > > > > > > > > > > > > templates.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > we MUST come up with a solution to
> > deprecate
> > > > '
> > > > > > > > > > > > > download.cloud.com
> > > > > > > > > > > > > > ',
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > is going to be going away any day now.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > i don't know the right way to solve this
> > to
> > > > be
> > > > > > > > honest,
> > > > > > > > > > but
> > > > > > > > > > > if
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > ideas, i am willing to help.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan
> > > > Johnson
> > > > > > <
> > > > > > > > > > > > > > njohnson@ena.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > So not to re-open a can of worms, but
> > I’m
> > > > in a
> > > > > > > > > > situation
> > > > > > > > > > > > > where
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > > > > to come up with a Marvin component test
> > > > that
> > > > > > > > depends
> > > > > > > > > > on a
> > > > > > > > > > > > > > > template
> > > > > > > > > > > > > > > > > > > > based on a kernel that’s relatively
> > new,
> > > > i.e.,
> > > > > > > > newer
> > > > > > > > > > than
> > > > > > > > > > > > > > Centos
> > > > > > > > > > > > > > > > 5.3
> > > > > > > > > > > > > > > > > /
> > > > > > > > > > > > > > > > > > > Ubuntu 10.04 .
> > > > > > > > > > > > > > > > > > > > I see openvm.eu has a suitable
> > template
> > > > > > (Ubuntu
> > > > > > > > > 16.0.4
> > > > > > > > > > > for
> > > > > > > > > > > > > > KVM),
> > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > from looking at the thread "Migrating
> > > > > > CloudStack
> > > > > > > > > > content
> > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > download.cloud.com” it looks like
> > there is
> > > > > > > > > resistance
> > > > > > > > > > to
> > > > > > > > > > > > > using
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > > least for hosting system vm templates
> > over
> > > > > > > concerns
> > > > > > > > > of
> > > > > > > > > > > > > > > neutrality.
> > > > > > > > > > > > > > > > > > > > Would this be suitable for a component
> > > > test?
> > > > > > If
> > > > > > > > not,
> > > > > > > > > > > what
> > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > “blessed”
> > > > > > > > > > > > > > > > > > > > template location?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks in advance!
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Nathan Johnson
> > > > > > > > > > > > > > > > > > > > R&D Engineer
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > > > > > > > > > > > > Nashville, TN 37211
> > > > > > > > > > > > > > > > > > > > General Office: 615-312-6000
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > website | blog | support
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > DISCLAIMER
> > > > > > > > > > > > > > > > > > ==========
> > > > > > > > > > > > > > > > > > This e-mail may contain privileged and
> > > > confidential
> > > > > > > > > > > information
> > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > the property of Accelerite, a Persistent
> > > > Systems
> > > > > > > > > business.
> > > > > > > > > > It
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > intended
> > > > > > > > > > > > > > > > > > only for the use of the individual or
> > entity to
> > > > > > which
> > > > > > > > it
> > > > > > > > > is
> > > > > > > > > > > > > > > addressed.
> > > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > > you are not the intended recipient, you
> > are not
> > > > > > > > > authorized
> > > > > > > > > > to
> > > > > > > > > > > > > read,
> > > > > > > > > > > > > > > > > retain,
> > > > > > > > > > > > > > > > > > copy, print, distribute or use this
> > message.
> > > > If you
> > > > > > > > have
> > > > > > > > > > > > received
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > communication in error, please notify the
> > > > sender
> > > > > > and
> > > > > > > > > delete
> > > > > > > > > > > all
> > > > > > > > > > > > > > > copies
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > this message. Accelerite, a Persistent
> > Systems
> > > > > > > business
> > > > > > > > > > does
> > > > > > > > > > > > not
> > > > > > > > > > > > > > > accept
> > > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > liability for virus infected mails.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Rafael Weingärtner
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Rafael Weingärtner
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Rafael Weingärtner
> > > > > > >
> > > > > >
> > > >
> >

Re: Modern template hosting

Posted by Chiradeep Vittal <ch...@gmail.com>.

Yes, that's my preferred solution. But there's a couple of scenarios to be
dealt with:
 1. standard isolated networks where the VR is the default router
 2. basic zone where the VR is not the default router
 3. isolated networks where the VR is not the default router

For (2) and (3) you may have to send down a static route in the DHCP
response to point 169.254.169.254 to the VR (or do it in all cases,
shouldn't be a big deal).


On Tue, Feb 28, 2017 at 9:57 AM, Wido den Hollander <wi...@widodh.nl> wrote:

>
> > Op 28 februari 2017 om 18:55 schreef Chiradeep Vittal <
> chiradeepv@gmail.com>:
> >
> >
> > To repeat what was discussed earlier in the thread: the cloud templates
> > provided by Ubuntu and CentOS do not work because they expect the
> metadata
> > URL to be at http://169.254.269.254/. Well, they work, but they take
> > awfully long to time out and after they time out, you still wouldn't  be
> > able to login since they do not install the ssh keys.
>
> But that is a 'simple fix' in the VR. Like you mentioned, add a DNAT rule
> in the VR and have the VR handle the traffic.
>
> Wido
>
> >
> > On Tue, Feb 28, 2017 at 9:34 AM, Wido den Hollander <wi...@widodh.nl>
> wrote:
> >
> > >
> > > > Op 28 februari 2017 om 2:40 schreef Pierre-Luc Dion <
> pdion891@apache.org
> > > >:
> > > >
> > > >
> > > > hi!
> > > >
> > > > I think we should work with distro provider to have their cloud
> builds
> > > work
> > > > with cloudstack. Good example is CoreOS, it work out of the box from
> > > their
> > > > channel builds.
> > > > it shouldn't be too complicated to have centos, ubuntu and debian,
> unless
> > > > ...
> > >
> > > I fully agree. I think that we shouldn't bother about VM templates.
> CentOS
> > > and Ubuntu both have a proper template which we can include.
> > >
> > > The less we host as a project, the better.
> > >
> > > Keep in mind that we host a Open Source project. the RPM and DEB
> packages
> > > on cloudstack.apt-get.eu are a service, but we release source as a
> > > project.
> > >
> > > Wido
> > >
> > > >
> > > > For our systemvm templates, can we just change the URL for somthing
> like
> > > > cloudstack.apache.org/systemvm/... ?
> > > > and for old systemvm that are depricated but that we might want to
> keep,
> > > > could we archive them into a github repo in
> > > > https://github.com/apachecloudstack ?
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Feb 27, 2017 at 6:28 PM, Chiradeep Vittal <
> chiradeepv@gmail.com>
> > > > wrote:
> > > >
> > > > > My stance is that the current workflow does a disservice to the
> user
> > > > > community by letting them install / use outdated and insecure
> > > templates.
> > > > > Now, let's assume download.cloud.com is gone forever. What do we
> tell
> > > ACS
> > > > > users pre-4.11 as far as *built-in templates* go?
> > > > > 1. Direct them to update templates.sql with some new URL, but with
> the
> > > same
> > > > > dirty old templates
> > > > > 2. Direct them to update templates.sql with some new URL, but with
> nice
> > > > > templates (e.g., open.vm.eu)
> > > > > 3. Same as (2), but document more choices.
> > > > >
> > > > > Now, why should things be different for 4.11 and later?
> Documenting the
> > > > > steps to install templates offline is trivial (and can be scripted
> to a
> > > > > large part, like cloud-install-sys-tmplt)
> > > > >
> > > > > For pre-4.11 users, for *systemvms*, anyway we tell them to use
> > > > > http://cloudstack.apt-get.eu which is not controlled by ACS.
> > > > >
> > > > >
> > > > > On Mon, Feb 27, 2017 at 2:50 PM, Rafael Weingärtner <
> > > > > rafaelweingartner@gmail.com> wrote:
> > > > >
> > > > > > Agree with you.
> > > > > > We need to support the current working flow. And then, define the
> > > first
> > > > > > version that will start using the new approach.
> > > > > >
> > > > > > On Mon, Feb 27, 2017 at 5:36 PM, Will Stevens <
> wstevens@cloudops.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I think we almost need a two pronged approach.
> > > > > > >
> > > > > > > 1) Get a solution in place which will enable us to document and
> > > serve
> > > > > > > templates for legacy systems.  I will work on this.
> > > > > > > 2) Discuss and understand how we SHOULD be handling this
> problem
> > > in the
> > > > > > > future and in what release we can expect it.
> > > > > > >
> > > > > > > I think we need to do both.  I think we should start to try to
> > > really
> > > > > > > understand what we want to deliver in (2) going forward.
> > > > > > >
> > > > > > > *Will STEVENS*
> > > > > > > Lead Developer
> > > > > > >
> > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > >
> > > > > > > On Mon, Feb 27, 2017 at 4:53 PM, Rafael Weingärtner <
> > > > > > > rafaelweingartner@gmail.com> wrote:
> > > > > > >
> > > > > > > > My worry is exactly with system VMs templates.
> > > > > > > >
> > > > > > > > Currently, we indicate administrators to download them from
> > > > > > > > http://cloudstack.apt-get.eu/systemvm/4.6/ [1]. However, the
> > > > > > > installation
> > > > > > > > docs do not mention the expected hashes for the file that is
> > > going to
> > > > > > be
> > > > > > > > downloaded.
> > > > > > > > Also, I do not know the code that downloads system VMs
> templates
> > > > > (when
> > > > > > > > upgrading), but if the hash being checked is taken from the
> > > mirror
> > > > > used
> > > > > > > to
> > > > > > > > download the file; the only thing it checks is that if the
> > > download
> > > > > > > > finished successfully (no transmission errors). If we want to
> > > check
> > > > > > > > integrity, check that the template we created is untampered;
> we
> > > need
> > > > > to
> > > > > > > > host and serve the hash in a secure manner.
> > > > > > > >
> > > > > > > > [1]
> > > > > > > > http://docs.cloudstack.apache.org/projects/cloudstack-
> > > > > > > installation/en/4.9/
> > > > > > > > management-server/index.html#prepare-the-system-vm-template
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Feb 27, 2017 at 4:36 PM, Chiradeep Vittal <
> > > > > > chiradeepv@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hashes are checked (md5 IIRC) today.
> > > > > > > > > But given the issues, I think the project should steer away
> > > from
> > > > > > > hosting
> > > > > > > > > templates except the systemvm template.
> > > > > > > > >
> > > > > > > > > On Mon, Feb 27, 2017 at 1:31 PM, Rafael Weingärtner <
> > > > > > > > > rafaelweingartner@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Will, I think we could support different path structures.
> > > This
> > > > > can
> > > > > > > > > > facilitate different deployment of mirrors based on the
> > > structure
> > > > > > the
> > > > > > > > > host
> > > > > > > > > > has.
> > > > > > > > > >
> > > > > > > > > > Could I add something else to the discussion? Have we
> > > discussed
> > > > > the
> > > > > > > > > > security impacts of setting up this mirrors approach?
> > > > > > > > > > I mean, if any of the mirrors gets corrupted (let`s say
> by a
> > > > > > hacker),
> > > > > > > > and
> > > > > > > > > > the templates are injected with malicious code, an
> attacker
> > > could
> > > > > > > > > > potentially get un-monitored and unlimited access to a
> cloud
> > > > > > > > environment.
> > > > > > > > > >
> > > > > > > > > > If we assume that the mirror may get malicious (it is not
> > > that I
> > > > > do
> > > > > > > not
> > > > > > > > > > trust you guys, but bad things happen), we cannot host
> hashes
> > > > > > there.
> > > > > > > > > Where
> > > > > > > > > > do you think we could store Sha512 or another hash type
> for
> > > these
> > > > > > > > > > templates? Could we host in the newly proposed Github
> repo or
> > > > > maybe
> > > > > > > > some
> > > > > > > > > > place in the ACS website?
> > > > > > > > > >
> > > > > > > > > > This would have an impact on clients (needing clear
> > > > > documentation)
> > > > > > > and
> > > > > > > > > our
> > > > > > > > > > code that automatically downloads System VM templates
> (does
> > > it
> > > > > > check
> > > > > > > > > hashes
> > > > > > > > > > when automatically installing templates today? It may
> require
> > > > > > > > > > implementation changes).
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <
> > > > > > wstevens@cloudops.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > so this is what I am looking to do.  Please let me
> know if
> > > you
> > > > > > have
> > > > > > > > > > > suggestions for me or think I should be solving the
> > > problem a
> > > > > > > > different
> > > > > > > > > > > way.
> > > > > > > > > > >
> > > > > > > > > > > - We request a new Github repository from the ASF at:
> > > > > > > > > > > 'apache/cloudstack-mirror-list'
> > > > > > > > > > > - In this repository we track a text file in the
> 'gh-pages'
> > > > > > branch
> > > > > > > > > with a
> > > > > > > > > > > list of valid download mirrors.
> > > > > > > > > > > - I build a binary to be hosted by the ASF (or at least
> > > with
> > > > > the
> > > > > > > ASF
> > > > > > > > > > > pointing a domain at the binary and I could potentially
> > > host
> > > > > it).
> > > > > > > We
> > > > > > > > > > will
> > > > > > > > > > > see how they want to handle the hosting of the binary.
> > > > > > > > > > >
> > > > > > > > > > > The binary would expose a web server which would
> behave as
> > > > > > follows:
> > > > > > > > > > > - When the 'client' requests a download url the
> following
> > > flow
> > > > > is
> > > > > > > > > kicked
> > > > > > > > > > > off:
> > > > > > > > > > > -- The mirror list is queried from github (or from a
> static
> > > > > site
> > > > > > > > hosted
> > > > > > > > > > on
> > > > > > > > > > > asf, as we see fit).
> > > > > > > > > > > -- The Lat/Lon of the 'client' is determined based on
> > > their IP.
> > > > > > > > > > > -- The Lat/Lon for each of the 'mirror's is determined
> > > based on
> > > > > > an
> > > > > > > IP
> > > > > > > > > > > lookup of the hostname.
> > > > > > > > > > > -- The closest geographical mirror is determined, the
> > > target is
> > > > > > > > > validated
> > > > > > > > > > > to be available and the user is redirected.
> > > > > > > > > > >
> > > > > > > > > > > Some questions I have right now:
> > > > > > > > > > > - Will every mirror have the same path structure to
> access
> > > the
> > > > > > > > > equivalent
> > > > > > > > > > > resources?
> > > > > > > > > > > - Should we support adding a path to the mirror url to
> > > specify
> > > > > > the
> > > > > > > > path
> > > > > > > > > > to
> > > > > > > > > > > the base common path?
> > > > > > > > > > > -- Example: lets say the binary is hosted on '
> dl.acs.com'
> > > and
> > > > > > > there
> > > > > > > > > are
> > > > > > > > > > > three mirrors 'abc.com', 'pqr.com/files' and '
> > > > > xyx.com/downloads
> > > > > > '.
> > > > > > > > > > > -- If the path being requested is '
> > > > > > > > > > > dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it
> would
> > > > > result
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > following potential paths for the mirrors:
> > > > > > > > > > > -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > > > > > > -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > > > > > > -- 'xyz.com/downloads/templates/
> systemvm-4.6.xen.vhd.bz2'
> > > > > > > > > > >
> > > > > > > > > > > Does this all make sense?
> > > > > > > > > > >
> > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > Lead Developer
> > > > > > > > > > >
> > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <
> > > > > > > > > chiradeepv@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > My bad. A few lines down, this has been added
> recently:
> > > > > > > > > > > >
> > > > > > > > > > > > this.request.setFollowRedirects(true);
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <
> > > > > > > > > > williamstevens@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > OK. Thanks for the heads up.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > > > > > chiradeepv@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Sounds workable. The downloader code in the SSVM
> > > won't
> > > > > > follow
> > > > > > > > > > > > redirects I
> > > > > > > > > > > > > > think.
> > > > > > > > > > > > > > https://github.com/apache/cloudstack/blob/
> > > > > > > > > > > > 5511065fc20787619d9cd0444a65a3
> > > > > > > > > > > > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > > > > > > > > > > > HttpTemplateDownloader.java#L93
> > > > > > > > > > > > > > https://goo.gl/dSi0r5
> > > > > > > > > > > > > >  Might need to add
> > > > > > > > > > > > > > client.setRedirectStrategy(new
> > > LaxRedirectStrategy());
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <
> > > > > > > > > > wstevens@cloudops.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > We haven't opened a ticket yet because we don't
> > > have a
> > > > > > > > strategy
> > > > > > > > > > > yet.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > What do you guys think of this:
> > > > > > > > > > > > > > > - We setup a new github repo in the 'apache'
> org
> > > which
> > > > > > > > consists
> > > > > > > > > > of
> > > > > > > > > > > a
> > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > file with a list of active/supported mirrors.
> > > > > > > > > > > > > > > - I write a small web server, distributed as a
> > > binary,
> > > > > > > which
> > > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > > > > > > hosted
> > > > > > > > > > > > > > > by ASF Infra.  This web server will query the
> > > current
> > > > > > list
> > > > > > > of
> > > > > > > > > > > mirrors
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > will select one and then do a 302 redirect to
> that
> > > > > > mirror.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The act of 'choosing' a mirror could be done
> in a
> > > > > number
> > > > > > of
> > > > > > > > > ways.
> > > > > > > > > > > > > > > - If we want to define an order, then it could
> > > just try
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > top
> > > > > > > > > > > > of
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > list and work its way down.  It would curl the
> > > target
> > > > > to
> > > > > > > make
> > > > > > > > > > sure
> > > > > > > > > > > it
> > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > a 200 and if it does, it would do a 302
> redirect.
> > > > > > > > > > > > > > > - Or, if we want to distribute the load across
> the
> > > > > > mirrors,
> > > > > > > > we
> > > > > > > > > > > could
> > > > > > > > > > > > > pick
> > > > > > > > > > > > > > > from the list randomly.  Again, doing a curl to
> > > verify
> > > > > > the
> > > > > > > > > mirror
> > > > > > > > > > > is
> > > > > > > > > > > > up
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > then doing a redirect.
> > > > > > > > > > > > > > > - If we want to get fancy, we could do a
> reverse IP
> > > > > > lookup
> > > > > > > > and
> > > > > > > > > > try
> > > > > > > > > > > to
> > > > > > > > > > > > > > match
> > > > > > > > > > > > > > > the requester with their closest geographical
> > > mirror.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thoughts?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep
> Vittal
> > > <
> > > > > > > > > > > > > chiradeepv@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > What steps are needed to set up a mirror?
> What
> > > does
> > > > > > Infra
> > > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > > > do?
> > > > > > > > > > > > > > Has
> > > > > > > > > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja
> Pullela <
> > > > > > > > > > > > > > > > raja.pullela@accelerite.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi will,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I believe, we didn’t get to close ‘getting
> a
> > > mirror
> > > > > > on
> > > > > > > > > > Apache’
> > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > needed someone on the Apache Infra side to
> > > close
> > > > > > this.
> > > > > > > > BTW,
> > > > > > > > > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages
> > > this?)
> > > > > > has
> > > > > > > > > > all/most
> > > > > > > > > > > > of
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > content.  Once we can close on the Apache
> > > mirror
> > > > > for
> > > > > > > > > hosting
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > content, I
> > > > > > > > > > > > > > > > > can help assist getting the content there.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > For now, we have replicated the
> > > download.cloud.com
> > > > > > > > content
> > > > > > > > > > to
> > > > > > > > > > > ‘
> > > > > > > > > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > > > > > > > > Also, we are working on a set of
> > > steps/procedure to
> > > > > > > help
> > > > > > > > > with
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > change.  I will update everyone in about a
> > > week’s
> > > > > > time
> > > > > > > on
> > > > > > > > > the
> > > > > > > > > > > > > > details.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > Raja Pullela
> > > > > > > > > > > > > > > > > Engineering Team,
> > > > > > > > > > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On 2/24/17, 11:23 PM, "
> > > williamstevens@gmail.com on
> > > > > > > > behalf
> > > > > > > > > of
> > > > > > > > > > > > Will
> > > > > > > > > > > > > > > > > Stevens" <williamstevens@gmail.com on
> behalf
> > > of
> > > > > > > > > > > > > > wstevens@cloudops.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > unfortunately the template mirror
> conversation
> > > got
> > > > > > > caught
> > > > > > > > > up
> > > > > > > > > > in
> > > > > > > > > > > > > > details
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > nobody took the lead on implementing a
> > > solution.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > citrix has been pinging me every couple
> months
> > > to
> > > > > say
> > > > > > > > > 'dude,
> > > > > > > > > > we
> > > > > > > > > > > > > need
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > remove the dependency on
> download.citrix.com',
> > > > > but i
> > > > > > > > have
> > > > > > > > > > not
> > > > > > > > > > > > had
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > cycles to get in and solve the problem.
> the
> > > > > shutdown
> > > > > > > of
> > > > > > > > > that
> > > > > > > > > > > is
> > > > > > > > > > > > > > > imminent
> > > > > > > > > > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul
> Angus <
> > > > > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Nathan,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Ideally, if you put the template
> location in
> > > (or
> > > > > > use
> > > > > > > a
> > > > > > > > > > > template
> > > > > > > > > > > > > > > defined
> > > > > > > > > > > > > > > > > > in)  test_data.py then the actual
> location
> > > can be
> > > > > > > > > > overridden
> > > > > > > > > > > by
> > > > > > > > > > > > > > > anyone
> > > > > > > > > > > > > > > > > > testing.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > For Trillian, we've copied all of the
> > > templates
> > > > > > that
> > > > > > > > > people
> > > > > > > > > > > > have
> > > > > > > > > > > > > > > define
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > a local repo and then replace the URLs in
> > > > > > > test_data.py
> > > > > > > > to
> > > > > > > > > > > > reduce
> > > > > > > > > > > > > > > > > bandwidth
> > > > > > > > > > > > > > > > > > use and download times.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Ie:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >             "bootableIso":
> > > > > > > > > > > > > > > > > >                 {
> > > > > > > > > > > > > > > > > >                     "displaytext": "Test
> > > Bootable
> > > > > > > ISO",
> > > > > > > > > > > > > > > > > >                     "name": "testISO",
> > > > > > > > > > > > > > > > > >                     "bootable": True,
> > > > > > > > > > > > > > > > > >                     "ispublic": False,
> > > > > > > > > > > > > > > > > >                     "url": "{{
> > > > > > marvin_images_location
> > > > > > > > > > > > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > > > > > > > > > > >                     "ostype": 'Other
> Linux
> > > > > > (64-bit)',
> > > > > > > > > > > > > > > > > >                     "mode":
> 'HTTP_DOWNLOAD'
> > > > > > > > > > > > > > > > > >         },
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I thought that we had come up with a
> > > solution for
> > > > > > > > > > > > > > download.cloud.com
> > > > > > > > > > > > > > > ,
> > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > having a mirrorlist hosted in Community
> > > Apache
> > > > > > > 'space'
> > > > > > > > > with
> > > > > > > > > > > > > anyone
> > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > out themselves forward as a mirror.
> > > > > > > > > > > > > > > > > > But I must admit I lost track of whether
> > > anyone
> > > > > > made
> > > > > > > > the
> > > > > > > > > > > > > requisite
> > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > in code....
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Kind regards,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Paul Angus
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > > > > > > > > www.shapeblue.com
> > > > > > > > > > > > > > > > > > 53 Chandos Place, Covent Garden, London
> WC2N
> > > > > 4HSUK
> > > > > > > > > > > > > > > > > > @shapeblue
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > > > > > > > From: williamstevens@gmail.com [mailto:
> > > > > > > > > > williamstevens@gmail.
> > > > > > > > > > > > com]
> > > > > > > > > > > > > > On
> > > > > > > > > > > > > > > > > > Behalf Of Will Stevens
> > > > > > > > > > > > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > > > > > > > > > > Subject: Re: Modern template hosting
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > this is a hard questions.  in general, we
> > > should
> > > > > be
> > > > > > > > > setting
> > > > > > > > > > > up
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > mirror
> > > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > some cloudstack/apache domain and then
> > > mirror to
> > > > > > > other
> > > > > > > > > > > provided
> > > > > > > > > > > > > > > > > templates.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > we MUST come up with a solution to
> deprecate
> > > '
> > > > > > > > > > > > download.cloud.com
> > > > > > > > > > > > > ',
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > is going to be going away any day now.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > i don't know the right way to solve this
> to
> > > be
> > > > > > > honest,
> > > > > > > > > but
> > > > > > > > > > if
> > > > > > > > > > > > you
> > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > ideas, i am willing to help.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan
> > > Johnson
> > > > > <
> > > > > > > > > > > > > njohnson@ena.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > So not to re-open a can of worms, but
> I’m
> > > in a
> > > > > > > > > situation
> > > > > > > > > > > > where
> > > > > > > > > > > > > I
> > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > > > to come up with a Marvin component test
> > > that
> > > > > > > depends
> > > > > > > > > on a
> > > > > > > > > > > > > > template
> > > > > > > > > > > > > > > > > > > based on a kernel that’s relatively
> new,
> > > i.e.,
> > > > > > > newer
> > > > > > > > > than
> > > > > > > > > > > > > Centos
> > > > > > > > > > > > > > > 5.3
> > > > > > > > > > > > > > > > /
> > > > > > > > > > > > > > > > > > Ubuntu 10.04 .
> > > > > > > > > > > > > > > > > > > I see openvm.eu has a suitable
> template
> > > > > (Ubuntu
> > > > > > > > 16.0.4
> > > > > > > > > > for
> > > > > > > > > > > > > KVM),
> > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > from looking at the thread "Migrating
> > > > > CloudStack
> > > > > > > > > content
> > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > download.cloud.com” it looks like
> there is
> > > > > > > > resistance
> > > > > > > > > to
> > > > > > > > > > > > using
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > least for hosting system vm templates
> over
> > > > > > concerns
> > > > > > > > of
> > > > > > > > > > > > > > neutrality.
> > > > > > > > > > > > > > > > > > > Would this be suitable for a component
> > > test?
> > > > > If
> > > > > > > not,
> > > > > > > > > > what
> > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > “blessed”
> > > > > > > > > > > > > > > > > > > template location?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks in advance!
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Nathan Johnson
> > > > > > > > > > > > > > > > > > > R&D Engineer
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > > > > > > > > > > > Nashville, TN 37211
> > > > > > > > > > > > > > > > > > > General Office: 615-312-6000
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > website | blog | support
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > DISCLAIMER
> > > > > > > > > > > > > > > > > ==========
> > > > > > > > > > > > > > > > > This e-mail may contain privileged and
> > > confidential
> > > > > > > > > > information
> > > > > > > > > > > > > which
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > the property of Accelerite, a Persistent
> > > Systems
> > > > > > > > business.
> > > > > > > > > It
> > > > > > > > > > > is
> > > > > > > > > > > > > > > intended
> > > > > > > > > > > > > > > > > only for the use of the individual or
> entity to
> > > > > which
> > > > > > > it
> > > > > > > > is
> > > > > > > > > > > > > > addressed.
> > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > you are not the intended recipient, you
> are not
> > > > > > > > authorized
> > > > > > > > > to
> > > > > > > > > > > > read,
> > > > > > > > > > > > > > > > retain,
> > > > > > > > > > > > > > > > > copy, print, distribute or use this
> message.
> > > If you
> > > > > > > have
> > > > > > > > > > > received
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > communication in error, please notify the
> > > sender
> > > > > and
> > > > > > > > delete
> > > > > > > > > > all
> > > > > > > > > > > > > > copies
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > this message. Accelerite, a Persistent
> Systems
> > > > > > business
> > > > > > > > > does
> > > > > > > > > > > not
> > > > > > > > > > > > > > accept
> > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > liability for virus infected mails.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Rafael Weingärtner
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Rafael Weingärtner
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Rafael Weingärtner
> > > > > >
> > > > >
> > >
>

Re: Modern template hosting

Posted by Wido den Hollander <wi...@widodh.nl>.

> Op 28 februari 2017 om 18:55 schreef Chiradeep Vittal <ch...@gmail.com>:
> 
> 
> To repeat what was discussed earlier in the thread: the cloud templates
> provided by Ubuntu and CentOS do not work because they expect the metadata
> URL to be at http://169.254.269.254/. Well, they work, but they take
> awfully long to time out and after they time out, you still wouldn't  be
> able to login since they do not install the ssh keys.

But that is a 'simple fix' in the VR. Like you mentioned, add a DNAT rule in the VR and have the VR handle the traffic.

Wido

> 
> On Tue, Feb 28, 2017 at 9:34 AM, Wido den Hollander <wi...@widodh.nl> wrote:
> 
> >
> > > Op 28 februari 2017 om 2:40 schreef Pierre-Luc Dion <pdion891@apache.org
> > >:
> > >
> > >
> > > hi!
> > >
> > > I think we should work with distro provider to have their cloud builds
> > work
> > > with cloudstack. Good example is CoreOS, it work out of the box from
> > their
> > > channel builds.
> > > it shouldn't be too complicated to have centos, ubuntu and debian, unless
> > > ...
> >
> > I fully agree. I think that we shouldn't bother about VM templates. CentOS
> > and Ubuntu both have a proper template which we can include.
> >
> > The less we host as a project, the better.
> >
> > Keep in mind that we host a Open Source project. the RPM and DEB packages
> > on cloudstack.apt-get.eu are a service, but we release source as a
> > project.
> >
> > Wido
> >
> > >
> > > For our systemvm templates, can we just change the URL for somthing like
> > > cloudstack.apache.org/systemvm/... ?
> > > and for old systemvm that are depricated but that we might want to keep,
> > > could we archive them into a github repo in
> > > https://github.com/apachecloudstack ?
> > >
> > >
> > >
> > >
> > > On Mon, Feb 27, 2017 at 6:28 PM, Chiradeep Vittal <ch...@gmail.com>
> > > wrote:
> > >
> > > > My stance is that the current workflow does a disservice to the user
> > > > community by letting them install / use outdated and insecure
> > templates.
> > > > Now, let's assume download.cloud.com is gone forever. What do we tell
> > ACS
> > > > users pre-4.11 as far as *built-in templates* go?
> > > > 1. Direct them to update templates.sql with some new URL, but with the
> > same
> > > > dirty old templates
> > > > 2. Direct them to update templates.sql with some new URL, but with nice
> > > > templates (e.g., open.vm.eu)
> > > > 3. Same as (2), but document more choices.
> > > >
> > > > Now, why should things be different for 4.11 and later? Documenting the
> > > > steps to install templates offline is trivial (and can be scripted to a
> > > > large part, like cloud-install-sys-tmplt)
> > > >
> > > > For pre-4.11 users, for *systemvms*, anyway we tell them to use
> > > > http://cloudstack.apt-get.eu which is not controlled by ACS.
> > > >
> > > >
> > > > On Mon, Feb 27, 2017 at 2:50 PM, Rafael Weingärtner <
> > > > rafaelweingartner@gmail.com> wrote:
> > > >
> > > > > Agree with you.
> > > > > We need to support the current working flow. And then, define the
> > first
> > > > > version that will start using the new approach.
> > > > >
> > > > > On Mon, Feb 27, 2017 at 5:36 PM, Will Stevens <wstevens@cloudops.com
> > >
> > > > > wrote:
> > > > >
> > > > > > I think we almost need a two pronged approach.
> > > > > >
> > > > > > 1) Get a solution in place which will enable us to document and
> > serve
> > > > > > templates for legacy systems.  I will work on this.
> > > > > > 2) Discuss and understand how we SHOULD be handling this problem
> > in the
> > > > > > future and in what release we can expect it.
> > > > > >
> > > > > > I think we need to do both.  I think we should start to try to
> > really
> > > > > > understand what we want to deliver in (2) going forward.
> > > > > >
> > > > > > *Will STEVENS*
> > > > > > Lead Developer
> > > > > >
> > > > > > <https://goo.gl/NYZ8KK>
> > > > > >
> > > > > > On Mon, Feb 27, 2017 at 4:53 PM, Rafael Weingärtner <
> > > > > > rafaelweingartner@gmail.com> wrote:
> > > > > >
> > > > > > > My worry is exactly with system VMs templates.
> > > > > > >
> > > > > > > Currently, we indicate administrators to download them from
> > > > > > > http://cloudstack.apt-get.eu/systemvm/4.6/ [1]. However, the
> > > > > > installation
> > > > > > > docs do not mention the expected hashes for the file that is
> > going to
> > > > > be
> > > > > > > downloaded.
> > > > > > > Also, I do not know the code that downloads system VMs templates
> > > > (when
> > > > > > > upgrading), but if the hash being checked is taken from the
> > mirror
> > > > used
> > > > > > to
> > > > > > > download the file; the only thing it checks is that if the
> > download
> > > > > > > finished successfully (no transmission errors). If we want to
> > check
> > > > > > > integrity, check that the template we created is untampered; we
> > need
> > > > to
> > > > > > > host and serve the hash in a secure manner.
> > > > > > >
> > > > > > > [1]
> > > > > > > http://docs.cloudstack.apache.org/projects/cloudstack-
> > > > > > installation/en/4.9/
> > > > > > > management-server/index.html#prepare-the-system-vm-template
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Feb 27, 2017 at 4:36 PM, Chiradeep Vittal <
> > > > > chiradeepv@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hashes are checked (md5 IIRC) today.
> > > > > > > > But given the issues, I think the project should steer away
> > from
> > > > > > hosting
> > > > > > > > templates except the systemvm template.
> > > > > > > >
> > > > > > > > On Mon, Feb 27, 2017 at 1:31 PM, Rafael Weingärtner <
> > > > > > > > rafaelweingartner@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Will, I think we could support different path structures.
> > This
> > > > can
> > > > > > > > > facilitate different deployment of mirrors based on the
> > structure
> > > > > the
> > > > > > > > host
> > > > > > > > > has.
> > > > > > > > >
> > > > > > > > > Could I add something else to the discussion? Have we
> > discussed
> > > > the
> > > > > > > > > security impacts of setting up this mirrors approach?
> > > > > > > > > I mean, if any of the mirrors gets corrupted (let`s say by a
> > > > > hacker),
> > > > > > > and
> > > > > > > > > the templates are injected with malicious code, an attacker
> > could
> > > > > > > > > potentially get un-monitored and unlimited access to a cloud
> > > > > > > environment.
> > > > > > > > >
> > > > > > > > > If we assume that the mirror may get malicious (it is not
> > that I
> > > > do
> > > > > > not
> > > > > > > > > trust you guys, but bad things happen), we cannot host hashes
> > > > > there.
> > > > > > > > Where
> > > > > > > > > do you think we could store Sha512 or another hash type for
> > these
> > > > > > > > > templates? Could we host in the newly proposed Github repo or
> > > > maybe
> > > > > > > some
> > > > > > > > > place in the ACS website?
> > > > > > > > >
> > > > > > > > > This would have an impact on clients (needing clear
> > > > documentation)
> > > > > > and
> > > > > > > > our
> > > > > > > > > code that automatically downloads System VM templates (does
> > it
> > > > > check
> > > > > > > > hashes
> > > > > > > > > when automatically installing templates today? It may require
> > > > > > > > > implementation changes).
> > > > > > > > >
> > > > > > > > > On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <
> > > > > wstevens@cloudops.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > so this is what I am looking to do.  Please let me know if
> > you
> > > > > have
> > > > > > > > > > suggestions for me or think I should be solving the
> > problem a
> > > > > > > different
> > > > > > > > > > way.
> > > > > > > > > >
> > > > > > > > > > - We request a new Github repository from the ASF at:
> > > > > > > > > > 'apache/cloudstack-mirror-list'
> > > > > > > > > > - In this repository we track a text file in the 'gh-pages'
> > > > > branch
> > > > > > > > with a
> > > > > > > > > > list of valid download mirrors.
> > > > > > > > > > - I build a binary to be hosted by the ASF (or at least
> > with
> > > > the
> > > > > > ASF
> > > > > > > > > > pointing a domain at the binary and I could potentially
> > host
> > > > it).
> > > > > > We
> > > > > > > > > will
> > > > > > > > > > see how they want to handle the hosting of the binary.
> > > > > > > > > >
> > > > > > > > > > The binary would expose a web server which would behave as
> > > > > follows:
> > > > > > > > > > - When the 'client' requests a download url the following
> > flow
> > > > is
> > > > > > > > kicked
> > > > > > > > > > off:
> > > > > > > > > > -- The mirror list is queried from github (or from a static
> > > > site
> > > > > > > hosted
> > > > > > > > > on
> > > > > > > > > > asf, as we see fit).
> > > > > > > > > > -- The Lat/Lon of the 'client' is determined based on
> > their IP.
> > > > > > > > > > -- The Lat/Lon for each of the 'mirror's is determined
> > based on
> > > > > an
> > > > > > IP
> > > > > > > > > > lookup of the hostname.
> > > > > > > > > > -- The closest geographical mirror is determined, the
> > target is
> > > > > > > > validated
> > > > > > > > > > to be available and the user is redirected.
> > > > > > > > > >
> > > > > > > > > > Some questions I have right now:
> > > > > > > > > > - Will every mirror have the same path structure to access
> > the
> > > > > > > > equivalent
> > > > > > > > > > resources?
> > > > > > > > > > - Should we support adding a path to the mirror url to
> > specify
> > > > > the
> > > > > > > path
> > > > > > > > > to
> > > > > > > > > > the base common path?
> > > > > > > > > > -- Example: lets say the binary is hosted on 'dl.acs.com'
> > and
> > > > > > there
> > > > > > > > are
> > > > > > > > > > three mirrors 'abc.com', 'pqr.com/files' and '
> > > > xyx.com/downloads
> > > > > '.
> > > > > > > > > > -- If the path being requested is '
> > > > > > > > > > dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would
> > > > result
> > > > > in
> > > > > > > the
> > > > > > > > > > following potential paths for the mirrors:
> > > > > > > > > > -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > > > > > -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > > > > > -- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > > > > >
> > > > > > > > > > Does this all make sense?
> > > > > > > > > >
> > > > > > > > > > *Will STEVENS*
> > > > > > > > > > Lead Developer
> > > > > > > > > >
> > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <
> > > > > > > > chiradeepv@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > My bad. A few lines down, this has been added recently:
> > > > > > > > > > >
> > > > > > > > > > > this.request.setFollowRedirects(true);
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <
> > > > > > > > > williamstevens@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > OK. Thanks for the heads up.
> > > > > > > > > > > >
> > > > > > > > > > > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > > > > chiradeepv@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Sounds workable. The downloader code in the SSVM
> > won't
> > > > > follow
> > > > > > > > > > > redirects I
> > > > > > > > > > > > > think.
> > > > > > > > > > > > > https://github.com/apache/cloudstack/blob/
> > > > > > > > > > > 5511065fc20787619d9cd0444a65a3
> > > > > > > > > > > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > > > > > > > > > > HttpTemplateDownloader.java#L93
> > > > > > > > > > > > > https://goo.gl/dSi0r5
> > > > > > > > > > > > >  Might need to add
> > > > > > > > > > > > > client.setRedirectStrategy(new
> > LaxRedirectStrategy());
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <
> > > > > > > > > wstevens@cloudops.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > We haven't opened a ticket yet because we don't
> > have a
> > > > > > > strategy
> > > > > > > > > > yet.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > What do you guys think of this:
> > > > > > > > > > > > > > - We setup a new github repo in the 'apache' org
> > which
> > > > > > > consists
> > > > > > > > > of
> > > > > > > > > > a
> > > > > > > > > > > > > single
> > > > > > > > > > > > > > file with a list of active/supported mirrors.
> > > > > > > > > > > > > > - I write a small web server, distributed as a
> > binary,
> > > > > > which
> > > > > > > > can
> > > > > > > > > be
> > > > > > > > > > > > > hosted
> > > > > > > > > > > > > > by ASF Infra.  This web server will query the
> > current
> > > > > list
> > > > > > of
> > > > > > > > > > mirrors
> > > > > > > > > > > > and
> > > > > > > > > > > > > > will select one and then do a 302 redirect to that
> > > > > mirror.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The act of 'choosing' a mirror could be done in a
> > > > number
> > > > > of
> > > > > > > > ways.
> > > > > > > > > > > > > > - If we want to define an order, then it could
> > just try
> > > > > > from
> > > > > > > > the
> > > > > > > > > > top
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > list and work its way down.  It would curl the
> > target
> > > > to
> > > > > > make
> > > > > > > > > sure
> > > > > > > > > > it
> > > > > > > > > > > > > gets
> > > > > > > > > > > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > > > > > > > > > > - Or, if we want to distribute the load across the
> > > > > mirrors,
> > > > > > > we
> > > > > > > > > > could
> > > > > > > > > > > > pick
> > > > > > > > > > > > > > from the list randomly.  Again, doing a curl to
> > verify
> > > > > the
> > > > > > > > mirror
> > > > > > > > > > is
> > > > > > > > > > > up
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > then doing a redirect.
> > > > > > > > > > > > > > - If we want to get fancy, we could do a reverse IP
> > > > > lookup
> > > > > > > and
> > > > > > > > > try
> > > > > > > > > > to
> > > > > > > > > > > > > match
> > > > > > > > > > > > > > the requester with their closest geographical
> > mirror.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thoughts?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal
> > <
> > > > > > > > > > > > chiradeepv@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > What steps are needed to set up a mirror? What
> > does
> > > > > Infra
> > > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > do?
> > > > > > > > > > > > > Has
> > > > > > > > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > > > > > > > > raja.pullela@accelerite.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi will,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I believe, we didn’t get to close ‘getting a
> > mirror
> > > > > on
> > > > > > > > > Apache’
> > > > > > > > > > > > > because
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > needed someone on the Apache Infra side to
> > close
> > > > > this.
> > > > > > > BTW,
> > > > > > > > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages
> > this?)
> > > > > has
> > > > > > > > > all/most
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > content.  Once we can close on the Apache
> > mirror
> > > > for
> > > > > > > > hosting
> > > > > > > > > > the
> > > > > > > > > > > > > > > content, I
> > > > > > > > > > > > > > > > can help assist getting the content there.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > For now, we have replicated the
> > download.cloud.com
> > > > > > > content
> > > > > > > > > to
> > > > > > > > > > ‘
> > > > > > > > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > > > > > > > Also, we are working on a set of
> > steps/procedure to
> > > > > > help
> > > > > > > > with
> > > > > > > > > > > this
> > > > > > > > > > > > > > > > change.  I will update everyone in about a
> > week’s
> > > > > time
> > > > > > on
> > > > > > > > the
> > > > > > > > > > > > > details.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Raja Pullela
> > > > > > > > > > > > > > > > Engineering Team,
> > > > > > > > > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On 2/24/17, 11:23 PM, "
> > williamstevens@gmail.com on
> > > > > > > behalf
> > > > > > > > of
> > > > > > > > > > > Will
> > > > > > > > > > > > > > > > Stevens" <williamstevens@gmail.com on behalf
> > of
> > > > > > > > > > > > > wstevens@cloudops.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > unfortunately the template mirror conversation
> > got
> > > > > > caught
> > > > > > > > up
> > > > > > > > > in
> > > > > > > > > > > > > details
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > nobody took the lead on implementing a
> > solution.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > citrix has been pinging me every couple months
> > to
> > > > say
> > > > > > > > 'dude,
> > > > > > > > > we
> > > > > > > > > > > > need
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > remove the dependency on download.citrix.com',
> > > > but i
> > > > > > > have
> > > > > > > > > not
> > > > > > > > > > > had
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > cycles to get in and solve the problem.  the
> > > > shutdown
> > > > > > of
> > > > > > > > that
> > > > > > > > > > is
> > > > > > > > > > > > > > imminent
> > > > > > > > > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Nathan,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Ideally, if you put the template location in
> > (or
> > > > > use
> > > > > > a
> > > > > > > > > > template
> > > > > > > > > > > > > > defined
> > > > > > > > > > > > > > > > > in)  test_data.py then the actual location
> > can be
> > > > > > > > > overridden
> > > > > > > > > > by
> > > > > > > > > > > > > > anyone
> > > > > > > > > > > > > > > > > testing.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > For Trillian, we've copied all of the
> > templates
> > > > > that
> > > > > > > > people
> > > > > > > > > > > have
> > > > > > > > > > > > > > define
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > a local repo and then replace the URLs in
> > > > > > test_data.py
> > > > > > > to
> > > > > > > > > > > reduce
> > > > > > > > > > > > > > > > bandwidth
> > > > > > > > > > > > > > > > > use and download times.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Ie:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >             "bootableIso":
> > > > > > > > > > > > > > > > >                 {
> > > > > > > > > > > > > > > > >                     "displaytext": "Test
> > Bootable
> > > > > > ISO",
> > > > > > > > > > > > > > > > >                     "name": "testISO",
> > > > > > > > > > > > > > > > >                     "bootable": True,
> > > > > > > > > > > > > > > > >                     "ispublic": False,
> > > > > > > > > > > > > > > > >                     "url": "{{
> > > > > marvin_images_location
> > > > > > > > > > > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > > > > > > > > > >                     "ostype": 'Other Linux
> > > > > (64-bit)',
> > > > > > > > > > > > > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > > > > > > > > > > > > >         },
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I thought that we had come up with a
> > solution for
> > > > > > > > > > > > > download.cloud.com
> > > > > > > > > > > > > > ,
> > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > having a mirrorlist hosted in Community
> > Apache
> > > > > > 'space'
> > > > > > > > with
> > > > > > > > > > > > anyone
> > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > out themselves forward as a mirror.
> > > > > > > > > > > > > > > > > But I must admit I lost track of whether
> > anyone
> > > > > made
> > > > > > > the
> > > > > > > > > > > > requisite
> > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > in code....
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Kind regards,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Paul Angus
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > > > > > > > www.shapeblue.com
> > > > > > > > > > > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N
> > > > 4HSUK
> > > > > > > > > > > > > > > > > @shapeblue
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > > > > > > From: williamstevens@gmail.com [mailto:
> > > > > > > > > williamstevens@gmail.
> > > > > > > > > > > com]
> > > > > > > > > > > > > On
> > > > > > > > > > > > > > > > > Behalf Of Will Stevens
> > > > > > > > > > > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > > > > > > > > > Subject: Re: Modern template hosting
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > this is a hard questions.  in general, we
> > should
> > > > be
> > > > > > > > setting
> > > > > > > > > > up
> > > > > > > > > > > a
> > > > > > > > > > > > > > mirror
> > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > some cloudstack/apache domain and then
> > mirror to
> > > > > > other
> > > > > > > > > > provided
> > > > > > > > > > > > > > > > templates.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > we MUST come up with a solution to deprecate
> > '
> > > > > > > > > > > download.cloud.com
> > > > > > > > > > > > ',
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > is going to be going away any day now.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > i don't know the right way to solve this to
> > be
> > > > > > honest,
> > > > > > > > but
> > > > > > > > > if
> > > > > > > > > > > you
> > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > ideas, i am willing to help.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan
> > Johnson
> > > > <
> > > > > > > > > > > > njohnson@ena.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > So not to re-open a can of worms, but I’m
> > in a
> > > > > > > > situation
> > > > > > > > > > > where
> > > > > > > > > > > > I
> > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > > to come up with a Marvin component test
> > that
> > > > > > depends
> > > > > > > > on a
> > > > > > > > > > > > > template
> > > > > > > > > > > > > > > > > > based on a kernel that’s relatively new,
> > i.e.,
> > > > > > newer
> > > > > > > > than
> > > > > > > > > > > > Centos
> > > > > > > > > > > > > > 5.3
> > > > > > > > > > > > > > > /
> > > > > > > > > > > > > > > > > Ubuntu 10.04 .
> > > > > > > > > > > > > > > > > > I see openvm.eu has a suitable template
> > > > (Ubuntu
> > > > > > > 16.0.4
> > > > > > > > > for
> > > > > > > > > > > > KVM),
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > from looking at the thread "Migrating
> > > > CloudStack
> > > > > > > > content
> > > > > > > > > > from
> > > > > > > > > > > > > > > > > > download.cloud.com” it looks like there is
> > > > > > > resistance
> > > > > > > > to
> > > > > > > > > > > using
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > least for hosting system vm templates over
> > > > > concerns
> > > > > > > of
> > > > > > > > > > > > > neutrality.
> > > > > > > > > > > > > > > > > > Would this be suitable for a component
> > test?
> > > > If
> > > > > > not,
> > > > > > > > > what
> > > > > > > > > > > is a
> > > > > > > > > > > > > > > > “blessed”
> > > > > > > > > > > > > > > > > > template location?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks in advance!
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Nathan Johnson
> > > > > > > > > > > > > > > > > > R&D Engineer
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > > > > > > > > > > Nashville, TN 37211
> > > > > > > > > > > > > > > > > > General Office: 615-312-6000
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > website | blog | support
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > DISCLAIMER
> > > > > > > > > > > > > > > > ==========
> > > > > > > > > > > > > > > > This e-mail may contain privileged and
> > confidential
> > > > > > > > > information
> > > > > > > > > > > > which
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > the property of Accelerite, a Persistent
> > Systems
> > > > > > > business.
> > > > > > > > It
> > > > > > > > > > is
> > > > > > > > > > > > > > intended
> > > > > > > > > > > > > > > > only for the use of the individual or entity to
> > > > which
> > > > > > it
> > > > > > > is
> > > > > > > > > > > > > addressed.
> > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > you are not the intended recipient, you are not
> > > > > > > authorized
> > > > > > > > to
> > > > > > > > > > > read,
> > > > > > > > > > > > > > > retain,
> > > > > > > > > > > > > > > > copy, print, distribute or use this message.
> > If you
> > > > > > have
> > > > > > > > > > received
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > communication in error, please notify the
> > sender
> > > > and
> > > > > > > delete
> > > > > > > > > all
> > > > > > > > > > > > > copies
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > this message. Accelerite, a Persistent Systems
> > > > > business
> > > > > > > > does
> > > > > > > > > > not
> > > > > > > > > > > > > accept
> > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > liability for virus infected mails.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Rafael Weingärtner
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Rafael Weingärtner
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rafael Weingärtner
> > > > >
> > > >
> >

Re: Modern template hosting

Posted by Chiradeep Vittal <ch...@gmail.com>.

To repeat what was discussed earlier in the thread: the cloud templates
provided by Ubuntu and CentOS do not work because they expect the metadata
URL to be at http://169.254.269.254/. Well, they work, but they take
awfully long to time out and after they time out, you still wouldn't  be
able to login since they do not install the ssh keys.

On Tue, Feb 28, 2017 at 9:34 AM, Wido den Hollander <wi...@widodh.nl> wrote:

>
> > Op 28 februari 2017 om 2:40 schreef Pierre-Luc Dion <pdion891@apache.org
> >:
> >
> >
> > hi!
> >
> > I think we should work with distro provider to have their cloud builds
> work
> > with cloudstack. Good example is CoreOS, it work out of the box from
> their
> > channel builds.
> > it shouldn't be too complicated to have centos, ubuntu and debian, unless
> > ...
>
> I fully agree. I think that we shouldn't bother about VM templates. CentOS
> and Ubuntu both have a proper template which we can include.
>
> The less we host as a project, the better.
>
> Keep in mind that we host a Open Source project. the RPM and DEB packages
> on cloudstack.apt-get.eu are a service, but we release source as a
> project.
>
> Wido
>
> >
> > For our systemvm templates, can we just change the URL for somthing like
> > cloudstack.apache.org/systemvm/... ?
> > and for old systemvm that are depricated but that we might want to keep,
> > could we archive them into a github repo in
> > https://github.com/apachecloudstack ?
> >
> >
> >
> >
> > On Mon, Feb 27, 2017 at 6:28 PM, Chiradeep Vittal <ch...@gmail.com>
> > wrote:
> >
> > > My stance is that the current workflow does a disservice to the user
> > > community by letting them install / use outdated and insecure
> templates.
> > > Now, let's assume download.cloud.com is gone forever. What do we tell
> ACS
> > > users pre-4.11 as far as *built-in templates* go?
> > > 1. Direct them to update templates.sql with some new URL, but with the
> same
> > > dirty old templates
> > > 2. Direct them to update templates.sql with some new URL, but with nice
> > > templates (e.g., open.vm.eu)
> > > 3. Same as (2), but document more choices.
> > >
> > > Now, why should things be different for 4.11 and later? Documenting the
> > > steps to install templates offline is trivial (and can be scripted to a
> > > large part, like cloud-install-sys-tmplt)
> > >
> > > For pre-4.11 users, for *systemvms*, anyway we tell them to use
> > > http://cloudstack.apt-get.eu which is not controlled by ACS.
> > >
> > >
> > > On Mon, Feb 27, 2017 at 2:50 PM, Rafael Weingärtner <
> > > rafaelweingartner@gmail.com> wrote:
> > >
> > > > Agree with you.
> > > > We need to support the current working flow. And then, define the
> first
> > > > version that will start using the new approach.
> > > >
> > > > On Mon, Feb 27, 2017 at 5:36 PM, Will Stevens <wstevens@cloudops.com
> >
> > > > wrote:
> > > >
> > > > > I think we almost need a two pronged approach.
> > > > >
> > > > > 1) Get a solution in place which will enable us to document and
> serve
> > > > > templates for legacy systems.  I will work on this.
> > > > > 2) Discuss and understand how we SHOULD be handling this problem
> in the
> > > > > future and in what release we can expect it.
> > > > >
> > > > > I think we need to do both.  I think we should start to try to
> really
> > > > > understand what we want to deliver in (2) going forward.
> > > > >
> > > > > *Will STEVENS*
> > > > > Lead Developer
> > > > >
> > > > > <https://goo.gl/NYZ8KK>
> > > > >
> > > > > On Mon, Feb 27, 2017 at 4:53 PM, Rafael Weingärtner <
> > > > > rafaelweingartner@gmail.com> wrote:
> > > > >
> > > > > > My worry is exactly with system VMs templates.
> > > > > >
> > > > > > Currently, we indicate administrators to download them from
> > > > > > http://cloudstack.apt-get.eu/systemvm/4.6/ [1]. However, the
> > > > > installation
> > > > > > docs do not mention the expected hashes for the file that is
> going to
> > > > be
> > > > > > downloaded.
> > > > > > Also, I do not know the code that downloads system VMs templates
> > > (when
> > > > > > upgrading), but if the hash being checked is taken from the
> mirror
> > > used
> > > > > to
> > > > > > download the file; the only thing it checks is that if the
> download
> > > > > > finished successfully (no transmission errors). If we want to
> check
> > > > > > integrity, check that the template we created is untampered; we
> need
> > > to
> > > > > > host and serve the hash in a secure manner.
> > > > > >
> > > > > > [1]
> > > > > > http://docs.cloudstack.apache.org/projects/cloudstack-
> > > > > installation/en/4.9/
> > > > > > management-server/index.html#prepare-the-system-vm-template
> > > > > >
> > > > > >
> > > > > > On Mon, Feb 27, 2017 at 4:36 PM, Chiradeep Vittal <
> > > > chiradeepv@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hashes are checked (md5 IIRC) today.
> > > > > > > But given the issues, I think the project should steer away
> from
> > > > > hosting
> > > > > > > templates except the systemvm template.
> > > > > > >
> > > > > > > On Mon, Feb 27, 2017 at 1:31 PM, Rafael Weingärtner <
> > > > > > > rafaelweingartner@gmail.com> wrote:
> > > > > > >
> > > > > > > > Will, I think we could support different path structures.
> This
> > > can
> > > > > > > > facilitate different deployment of mirrors based on the
> structure
> > > > the
> > > > > > > host
> > > > > > > > has.
> > > > > > > >
> > > > > > > > Could I add something else to the discussion? Have we
> discussed
> > > the
> > > > > > > > security impacts of setting up this mirrors approach?
> > > > > > > > I mean, if any of the mirrors gets corrupted (let`s say by a
> > > > hacker),
> > > > > > and
> > > > > > > > the templates are injected with malicious code, an attacker
> could
> > > > > > > > potentially get un-monitored and unlimited access to a cloud
> > > > > > environment.
> > > > > > > >
> > > > > > > > If we assume that the mirror may get malicious (it is not
> that I
> > > do
> > > > > not
> > > > > > > > trust you guys, but bad things happen), we cannot host hashes
> > > > there.
> > > > > > > Where
> > > > > > > > do you think we could store Sha512 or another hash type for
> these
> > > > > > > > templates? Could we host in the newly proposed Github repo or
> > > maybe
> > > > > > some
> > > > > > > > place in the ACS website?
> > > > > > > >
> > > > > > > > This would have an impact on clients (needing clear
> > > documentation)
> > > > > and
> > > > > > > our
> > > > > > > > code that automatically downloads System VM templates (does
> it
> > > > check
> > > > > > > hashes
> > > > > > > > when automatically installing templates today? It may require
> > > > > > > > implementation changes).
> > > > > > > >
> > > > > > > > On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <
> > > > wstevens@cloudops.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > so this is what I am looking to do.  Please let me know if
> you
> > > > have
> > > > > > > > > suggestions for me or think I should be solving the
> problem a
> > > > > > different
> > > > > > > > > way.
> > > > > > > > >
> > > > > > > > > - We request a new Github repository from the ASF at:
> > > > > > > > > 'apache/cloudstack-mirror-list'
> > > > > > > > > - In this repository we track a text file in the 'gh-pages'
> > > > branch
> > > > > > > with a
> > > > > > > > > list of valid download mirrors.
> > > > > > > > > - I build a binary to be hosted by the ASF (or at least
> with
> > > the
> > > > > ASF
> > > > > > > > > pointing a domain at the binary and I could potentially
> host
> > > it).
> > > > > We
> > > > > > > > will
> > > > > > > > > see how they want to handle the hosting of the binary.
> > > > > > > > >
> > > > > > > > > The binary would expose a web server which would behave as
> > > > follows:
> > > > > > > > > - When the 'client' requests a download url the following
> flow
> > > is
> > > > > > > kicked
> > > > > > > > > off:
> > > > > > > > > -- The mirror list is queried from github (or from a static
> > > site
> > > > > > hosted
> > > > > > > > on
> > > > > > > > > asf, as we see fit).
> > > > > > > > > -- The Lat/Lon of the 'client' is determined based on
> their IP.
> > > > > > > > > -- The Lat/Lon for each of the 'mirror's is determined
> based on
> > > > an
> > > > > IP
> > > > > > > > > lookup of the hostname.
> > > > > > > > > -- The closest geographical mirror is determined, the
> target is
> > > > > > > validated
> > > > > > > > > to be available and the user is redirected.
> > > > > > > > >
> > > > > > > > > Some questions I have right now:
> > > > > > > > > - Will every mirror have the same path structure to access
> the
> > > > > > > equivalent
> > > > > > > > > resources?
> > > > > > > > > - Should we support adding a path to the mirror url to
> specify
> > > > the
> > > > > > path
> > > > > > > > to
> > > > > > > > > the base common path?
> > > > > > > > > -- Example: lets say the binary is hosted on 'dl.acs.com'
> and
> > > > > there
> > > > > > > are
> > > > > > > > > three mirrors 'abc.com', 'pqr.com/files' and '
> > > xyx.com/downloads
> > > > '.
> > > > > > > > > -- If the path being requested is '
> > > > > > > > > dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would
> > > result
> > > > in
> > > > > > the
> > > > > > > > > following potential paths for the mirrors:
> > > > > > > > > -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > > > > -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > > > > -- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > > > >
> > > > > > > > > Does this all make sense?
> > > > > > > > >
> > > > > > > > > *Will STEVENS*
> > > > > > > > > Lead Developer
> > > > > > > > >
> > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > >
> > > > > > > > > On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <
> > > > > > > chiradeepv@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > My bad. A few lines down, this has been added recently:
> > > > > > > > > >
> > > > > > > > > > this.request.setFollowRedirects(true);
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <
> > > > > > > > williamstevens@gmail.com
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > OK. Thanks for the heads up.
> > > > > > > > > > >
> > > > > > > > > > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > > > chiradeepv@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Sounds workable. The downloader code in the SSVM
> won't
> > > > follow
> > > > > > > > > > redirects I
> > > > > > > > > > > > think.
> > > > > > > > > > > > https://github.com/apache/cloudstack/blob/
> > > > > > > > > > 5511065fc20787619d9cd0444a65a3
> > > > > > > > > > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > > > > > > > > > HttpTemplateDownloader.java#L93
> > > > > > > > > > > > https://goo.gl/dSi0r5
> > > > > > > > > > > >  Might need to add
> > > > > > > > > > > > client.setRedirectStrategy(new
> LaxRedirectStrategy());
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <
> > > > > > > > wstevens@cloudops.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > We haven't opened a ticket yet because we don't
> have a
> > > > > > strategy
> > > > > > > > > yet.
> > > > > > > > > > > > >
> > > > > > > > > > > > > What do you guys think of this:
> > > > > > > > > > > > > - We setup a new github repo in the 'apache' org
> which
> > > > > > consists
> > > > > > > > of
> > > > > > > > > a
> > > > > > > > > > > > single
> > > > > > > > > > > > > file with a list of active/supported mirrors.
> > > > > > > > > > > > > - I write a small web server, distributed as a
> binary,
> > > > > which
> > > > > > > can
> > > > > > > > be
> > > > > > > > > > > > hosted
> > > > > > > > > > > > > by ASF Infra.  This web server will query the
> current
> > > > list
> > > > > of
> > > > > > > > > mirrors
> > > > > > > > > > > and
> > > > > > > > > > > > > will select one and then do a 302 redirect to that
> > > > mirror.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The act of 'choosing' a mirror could be done in a
> > > number
> > > > of
> > > > > > > ways.
> > > > > > > > > > > > > - If we want to define an order, then it could
> just try
> > > > > from
> > > > > > > the
> > > > > > > > > top
> > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > > list and work its way down.  It would curl the
> target
> > > to
> > > > > make
> > > > > > > > sure
> > > > > > > > > it
> > > > > > > > > > > > gets
> > > > > > > > > > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > > > > > > > > > - Or, if we want to distribute the load across the
> > > > mirrors,
> > > > > > we
> > > > > > > > > could
> > > > > > > > > > > pick
> > > > > > > > > > > > > from the list randomly.  Again, doing a curl to
> verify
> > > > the
> > > > > > > mirror
> > > > > > > > > is
> > > > > > > > > > up
> > > > > > > > > > > > and
> > > > > > > > > > > > > then doing a redirect.
> > > > > > > > > > > > > - If we want to get fancy, we could do a reverse IP
> > > > lookup
> > > > > > and
> > > > > > > > try
> > > > > > > > > to
> > > > > > > > > > > > match
> > > > > > > > > > > > > the requester with their closest geographical
> mirror.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thoughts?
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > >
> > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal
> <
> > > > > > > > > > > chiradeepv@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > What steps are needed to set up a mirror? What
> does
> > > > Infra
> > > > > > > need
> > > > > > > > to
> > > > > > > > > > do?
> > > > > > > > > > > > Has
> > > > > > > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > > > > > > > raja.pullela@accelerite.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi will,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I believe, we didn’t get to close ‘getting a
> mirror
> > > > on
> > > > > > > > Apache’
> > > > > > > > > > > > because
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > needed someone on the Apache Infra side to
> close
> > > > this.
> > > > > > BTW,
> > > > > > > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages
> this?)
> > > > has
> > > > > > > > all/most
> > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > content.  Once we can close on the Apache
> mirror
> > > for
> > > > > > > hosting
> > > > > > > > > the
> > > > > > > > > > > > > > content, I
> > > > > > > > > > > > > > > can help assist getting the content there.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For now, we have replicated the
> download.cloud.com
> > > > > > content
> > > > > > > > to
> > > > > > > > > ‘
> > > > > > > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > > > > > > Also, we are working on a set of
> steps/procedure to
> > > > > help
> > > > > > > with
> > > > > > > > > > this
> > > > > > > > > > > > > > > change.  I will update everyone in about a
> week’s
> > > > time
> > > > > on
> > > > > > > the
> > > > > > > > > > > > details.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Raja Pullela
> > > > > > > > > > > > > > > Engineering Team,
> > > > > > > > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On 2/24/17, 11:23 PM, "
> williamstevens@gmail.com on
> > > > > > behalf
> > > > > > > of
> > > > > > > > > > Will
> > > > > > > > > > > > > > > Stevens" <williamstevens@gmail.com on behalf
> of
> > > > > > > > > > > > wstevens@cloudops.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > unfortunately the template mirror conversation
> got
> > > > > caught
> > > > > > > up
> > > > > > > > in
> > > > > > > > > > > > details
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > nobody took the lead on implementing a
> solution.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > citrix has been pinging me every couple months
> to
> > > say
> > > > > > > 'dude,
> > > > > > > > we
> > > > > > > > > > > need
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > remove the dependency on download.citrix.com',
> > > but i
> > > > > > have
> > > > > > > > not
> > > > > > > > > > had
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > cycles to get in and solve the problem.  the
> > > shutdown
> > > > > of
> > > > > > > that
> > > > > > > > > is
> > > > > > > > > > > > > imminent
> > > > > > > > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Nathan,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Ideally, if you put the template location in
> (or
> > > > use
> > > > > a
> > > > > > > > > template
> > > > > > > > > > > > > defined
> > > > > > > > > > > > > > > > in)  test_data.py then the actual location
> can be
> > > > > > > > overridden
> > > > > > > > > by
> > > > > > > > > > > > > anyone
> > > > > > > > > > > > > > > > testing.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > For Trillian, we've copied all of the
> templates
> > > > that
> > > > > > > people
> > > > > > > > > > have
> > > > > > > > > > > > > define
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > a local repo and then replace the URLs in
> > > > > test_data.py
> > > > > > to
> > > > > > > > > > reduce
> > > > > > > > > > > > > > > bandwidth
> > > > > > > > > > > > > > > > use and download times.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Ie:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >             "bootableIso":
> > > > > > > > > > > > > > > >                 {
> > > > > > > > > > > > > > > >                     "displaytext": "Test
> Bootable
> > > > > ISO",
> > > > > > > > > > > > > > > >                     "name": "testISO",
> > > > > > > > > > > > > > > >                     "bootable": True,
> > > > > > > > > > > > > > > >                     "ispublic": False,
> > > > > > > > > > > > > > > >                     "url": "{{
> > > > marvin_images_location
> > > > > > > > > > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > > > > > > > > >                     "ostype": 'Other Linux
> > > > (64-bit)',
> > > > > > > > > > > > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > > > > > > > > > > > >         },
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I thought that we had come up with a
> solution for
> > > > > > > > > > > > download.cloud.com
> > > > > > > > > > > > > ,
> > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > having a mirrorlist hosted in Community
> Apache
> > > > > 'space'
> > > > > > > with
> > > > > > > > > > > anyone
> > > > > > > > > > > > > able
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > out themselves forward as a mirror.
> > > > > > > > > > > > > > > > But I must admit I lost track of whether
> anyone
> > > > made
> > > > > > the
> > > > > > > > > > > requisite
> > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > in code....
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Kind regards,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Paul Angus
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > > > > > > www.shapeblue.com
> > > > > > > > > > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N
> > > 4HSUK
> > > > > > > > > > > > > > > > @shapeblue
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > > > > > From: williamstevens@gmail.com [mailto:
> > > > > > > > williamstevens@gmail.
> > > > > > > > > > com]
> > > > > > > > > > > > On
> > > > > > > > > > > > > > > > Behalf Of Will Stevens
> > > > > > > > > > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > > > > > > > > Subject: Re: Modern template hosting
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > this is a hard questions.  in general, we
> should
> > > be
> > > > > > > setting
> > > > > > > > > up
> > > > > > > > > > a
> > > > > > > > > > > > > mirror
> > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > some cloudstack/apache domain and then
> mirror to
> > > > > other
> > > > > > > > > provided
> > > > > > > > > > > > > > > templates.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > we MUST come up with a solution to deprecate
> '
> > > > > > > > > > download.cloud.com
> > > > > > > > > > > ',
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > is going to be going away any day now.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > i don't know the right way to solve this to
> be
> > > > > honest,
> > > > > > > but
> > > > > > > > if
> > > > > > > > > > you
> > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > ideas, i am willing to help.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan
> Johnson
> > > <
> > > > > > > > > > > njohnson@ena.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > So not to re-open a can of worms, but I’m
> in a
> > > > > > > situation
> > > > > > > > > > where
> > > > > > > > > > > I
> > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > to come up with a Marvin component test
> that
> > > > > depends
> > > > > > > on a
> > > > > > > > > > > > template
> > > > > > > > > > > > > > > > > based on a kernel that’s relatively new,
> i.e.,
> > > > > newer
> > > > > > > than
> > > > > > > > > > > Centos
> > > > > > > > > > > > > 5.3
> > > > > > > > > > > > > > /
> > > > > > > > > > > > > > > > Ubuntu 10.04 .
> > > > > > > > > > > > > > > > > I see openvm.eu has a suitable template
> > > (Ubuntu
> > > > > > 16.0.4
> > > > > > > > for
> > > > > > > > > > > KVM),
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > from looking at the thread "Migrating
> > > CloudStack
> > > > > > > content
> > > > > > > > > from
> > > > > > > > > > > > > > > > > download.cloud.com” it looks like there is
> > > > > > resistance
> > > > > > > to
> > > > > > > > > > using
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > least for hosting system vm templates over
> > > > concerns
> > > > > > of
> > > > > > > > > > > > neutrality.
> > > > > > > > > > > > > > > > > Would this be suitable for a component
> test?
> > > If
> > > > > not,
> > > > > > > > what
> > > > > > > > > > is a
> > > > > > > > > > > > > > > “blessed”
> > > > > > > > > > > > > > > > > template location?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks in advance!
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Nathan Johnson
> > > > > > > > > > > > > > > > > R&D Engineer
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > > > > > > > > > Nashville, TN 37211
> > > > > > > > > > > > > > > > > General Office: 615-312-6000
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > website | blog | support
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > DISCLAIMER
> > > > > > > > > > > > > > > ==========
> > > > > > > > > > > > > > > This e-mail may contain privileged and
> confidential
> > > > > > > > information
> > > > > > > > > > > which
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > the property of Accelerite, a Persistent
> Systems
> > > > > > business.
> > > > > > > It
> > > > > > > > > is
> > > > > > > > > > > > > intended
> > > > > > > > > > > > > > > only for the use of the individual or entity to
> > > which
> > > > > it
> > > > > > is
> > > > > > > > > > > > addressed.
> > > > > > > > > > > > > If
> > > > > > > > > > > > > > > you are not the intended recipient, you are not
> > > > > > authorized
> > > > > > > to
> > > > > > > > > > read,
> > > > > > > > > > > > > > retain,
> > > > > > > > > > > > > > > copy, print, distribute or use this message.
> If you
> > > > > have
> > > > > > > > > received
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > communication in error, please notify the
> sender
> > > and
> > > > > > delete
> > > > > > > > all
> > > > > > > > > > > > copies
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > this message. Accelerite, a Persistent Systems
> > > > business
> > > > > > > does
> > > > > > > > > not
> > > > > > > > > > > > accept
> > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > liability for virus infected mails.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Rafael Weingärtner
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Rafael Weingärtner
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Rafael Weingärtner
> > > >
> > >
>

Re: Modern template hosting

Posted by Wido den Hollander <wi...@widodh.nl>.

> Op 28 februari 2017 om 2:40 schreef Pierre-Luc Dion <pd...@apache.org>:
> 
> 
> hi!
> 
> I think we should work with distro provider to have their cloud builds work
> with cloudstack. Good example is CoreOS, it work out of the box from their
> channel builds.
> it shouldn't be too complicated to have centos, ubuntu and debian, unless
> ...

I fully agree. I think that we shouldn't bother about VM templates. CentOS and Ubuntu both have a proper template which we can include.

The less we host as a project, the better.

Keep in mind that we host a Open Source project. the RPM and DEB packages on cloudstack.apt-get.eu are a service, but we release source as a project.

Wido

> 
> For our systemvm templates, can we just change the URL for somthing like
> cloudstack.apache.org/systemvm/... ?
> and for old systemvm that are depricated but that we might want to keep,
> could we archive them into a github repo in
> https://github.com/apachecloudstack ?
> 
> 
> 
> 
> On Mon, Feb 27, 2017 at 6:28 PM, Chiradeep Vittal <ch...@gmail.com>
> wrote:
> 
> > My stance is that the current workflow does a disservice to the user
> > community by letting them install / use outdated and insecure templates.
> > Now, let's assume download.cloud.com is gone forever. What do we tell ACS
> > users pre-4.11 as far as *built-in templates* go?
> > 1. Direct them to update templates.sql with some new URL, but with the same
> > dirty old templates
> > 2. Direct them to update templates.sql with some new URL, but with nice
> > templates (e.g., open.vm.eu)
> > 3. Same as (2), but document more choices.
> >
> > Now, why should things be different for 4.11 and later? Documenting the
> > steps to install templates offline is trivial (and can be scripted to a
> > large part, like cloud-install-sys-tmplt)
> >
> > For pre-4.11 users, for *systemvms*, anyway we tell them to use
> > http://cloudstack.apt-get.eu which is not controlled by ACS.
> >
> >
> > On Mon, Feb 27, 2017 at 2:50 PM, Rafael Weingärtner <
> > rafaelweingartner@gmail.com> wrote:
> >
> > > Agree with you.
> > > We need to support the current working flow. And then, define the first
> > > version that will start using the new approach.
> > >
> > > On Mon, Feb 27, 2017 at 5:36 PM, Will Stevens <ws...@cloudops.com>
> > > wrote:
> > >
> > > > I think we almost need a two pronged approach.
> > > >
> > > > 1) Get a solution in place which will enable us to document and serve
> > > > templates for legacy systems.  I will work on this.
> > > > 2) Discuss and understand how we SHOULD be handling this problem in the
> > > > future and in what release we can expect it.
> > > >
> > > > I think we need to do both.  I think we should start to try to really
> > > > understand what we want to deliver in (2) going forward.
> > > >
> > > > *Will STEVENS*
> > > > Lead Developer
> > > >
> > > > <https://goo.gl/NYZ8KK>
> > > >
> > > > On Mon, Feb 27, 2017 at 4:53 PM, Rafael Weingärtner <
> > > > rafaelweingartner@gmail.com> wrote:
> > > >
> > > > > My worry is exactly with system VMs templates.
> > > > >
> > > > > Currently, we indicate administrators to download them from
> > > > > http://cloudstack.apt-get.eu/systemvm/4.6/ [1]. However, the
> > > > installation
> > > > > docs do not mention the expected hashes for the file that is going to
> > > be
> > > > > downloaded.
> > > > > Also, I do not know the code that downloads system VMs templates
> > (when
> > > > > upgrading), but if the hash being checked is taken from the mirror
> > used
> > > > to
> > > > > download the file; the only thing it checks is that if the download
> > > > > finished successfully (no transmission errors). If we want to check
> > > > > integrity, check that the template we created is untampered; we need
> > to
> > > > > host and serve the hash in a secure manner.
> > > > >
> > > > > [1]
> > > > > http://docs.cloudstack.apache.org/projects/cloudstack-
> > > > installation/en/4.9/
> > > > > management-server/index.html#prepare-the-system-vm-template
> > > > >
> > > > >
> > > > > On Mon, Feb 27, 2017 at 4:36 PM, Chiradeep Vittal <
> > > chiradeepv@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hashes are checked (md5 IIRC) today.
> > > > > > But given the issues, I think the project should steer away from
> > > > hosting
> > > > > > templates except the systemvm template.
> > > > > >
> > > > > > On Mon, Feb 27, 2017 at 1:31 PM, Rafael Weingärtner <
> > > > > > rafaelweingartner@gmail.com> wrote:
> > > > > >
> > > > > > > Will, I think we could support different path structures. This
> > can
> > > > > > > facilitate different deployment of mirrors based on the structure
> > > the
> > > > > > host
> > > > > > > has.
> > > > > > >
> > > > > > > Could I add something else to the discussion? Have we discussed
> > the
> > > > > > > security impacts of setting up this mirrors approach?
> > > > > > > I mean, if any of the mirrors gets corrupted (let`s say by a
> > > hacker),
> > > > > and
> > > > > > > the templates are injected with malicious code, an attacker could
> > > > > > > potentially get un-monitored and unlimited access to a cloud
> > > > > environment.
> > > > > > >
> > > > > > > If we assume that the mirror may get malicious (it is not that I
> > do
> > > > not
> > > > > > > trust you guys, but bad things happen), we cannot host hashes
> > > there.
> > > > > > Where
> > > > > > > do you think we could store Sha512 or another hash type for these
> > > > > > > templates? Could we host in the newly proposed Github repo or
> > maybe
> > > > > some
> > > > > > > place in the ACS website?
> > > > > > >
> > > > > > > This would have an impact on clients (needing clear
> > documentation)
> > > > and
> > > > > > our
> > > > > > > code that automatically downloads System VM templates (does it
> > > check
> > > > > > hashes
> > > > > > > when automatically installing templates today? It may require
> > > > > > > implementation changes).
> > > > > > >
> > > > > > > On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <
> > > wstevens@cloudops.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > so this is what I am looking to do.  Please let me know if you
> > > have
> > > > > > > > suggestions for me or think I should be solving the problem a
> > > > > different
> > > > > > > > way.
> > > > > > > >
> > > > > > > > - We request a new Github repository from the ASF at:
> > > > > > > > 'apache/cloudstack-mirror-list'
> > > > > > > > - In this repository we track a text file in the 'gh-pages'
> > > branch
> > > > > > with a
> > > > > > > > list of valid download mirrors.
> > > > > > > > - I build a binary to be hosted by the ASF (or at least with
> > the
> > > > ASF
> > > > > > > > pointing a domain at the binary and I could potentially host
> > it).
> > > > We
> > > > > > > will
> > > > > > > > see how they want to handle the hosting of the binary.
> > > > > > > >
> > > > > > > > The binary would expose a web server which would behave as
> > > follows:
> > > > > > > > - When the 'client' requests a download url the following flow
> > is
> > > > > > kicked
> > > > > > > > off:
> > > > > > > > -- The mirror list is queried from github (or from a static
> > site
> > > > > hosted
> > > > > > > on
> > > > > > > > asf, as we see fit).
> > > > > > > > -- The Lat/Lon of the 'client' is determined based on their IP.
> > > > > > > > -- The Lat/Lon for each of the 'mirror's is determined based on
> > > an
> > > > IP
> > > > > > > > lookup of the hostname.
> > > > > > > > -- The closest geographical mirror is determined, the target is
> > > > > > validated
> > > > > > > > to be available and the user is redirected.
> > > > > > > >
> > > > > > > > Some questions I have right now:
> > > > > > > > - Will every mirror have the same path structure to access the
> > > > > > equivalent
> > > > > > > > resources?
> > > > > > > > - Should we support adding a path to the mirror url to specify
> > > the
> > > > > path
> > > > > > > to
> > > > > > > > the base common path?
> > > > > > > > -- Example: lets say the binary is hosted on 'dl.acs.com' and
> > > > there
> > > > > > are
> > > > > > > > three mirrors 'abc.com', 'pqr.com/files' and '
> > xyx.com/downloads
> > > '.
> > > > > > > > -- If the path being requested is '
> > > > > > > > dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would
> > result
> > > in
> > > > > the
> > > > > > > > following potential paths for the mirrors:
> > > > > > > > -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > > > -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > > > -- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > > >
> > > > > > > > Does this all make sense?
> > > > > > > >
> > > > > > > > *Will STEVENS*
> > > > > > > > Lead Developer
> > > > > > > >
> > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > >
> > > > > > > > On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <
> > > > > > chiradeepv@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > My bad. A few lines down, this has been added recently:
> > > > > > > > >
> > > > > > > > > this.request.setFollowRedirects(true);
> > > > > > > > >
> > > > > > > > > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <
> > > > > > > williamstevens@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > OK. Thanks for the heads up.
> > > > > > > > > >
> > > > > > > > > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > > chiradeepv@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Sounds workable. The downloader code in the SSVM won't
> > > follow
> > > > > > > > > redirects I
> > > > > > > > > > > think.
> > > > > > > > > > > https://github.com/apache/cloudstack/blob/
> > > > > > > > > 5511065fc20787619d9cd0444a65a3
> > > > > > > > > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > > > > > > > > HttpTemplateDownloader.java#L93
> > > > > > > > > > > https://goo.gl/dSi0r5
> > > > > > > > > > >  Might need to add
> > > > > > > > > > > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <
> > > > > > > wstevens@cloudops.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > We haven't opened a ticket yet because we don't have a
> > > > > strategy
> > > > > > > > yet.
> > > > > > > > > > > >
> > > > > > > > > > > > What do you guys think of this:
> > > > > > > > > > > > - We setup a new github repo in the 'apache' org which
> > > > > consists
> > > > > > > of
> > > > > > > > a
> > > > > > > > > > > single
> > > > > > > > > > > > file with a list of active/supported mirrors.
> > > > > > > > > > > > - I write a small web server, distributed as a binary,
> > > > which
> > > > > > can
> > > > > > > be
> > > > > > > > > > > hosted
> > > > > > > > > > > > by ASF Infra.  This web server will query the current
> > > list
> > > > of
> > > > > > > > mirrors
> > > > > > > > > > and
> > > > > > > > > > > > will select one and then do a 302 redirect to that
> > > mirror.
> > > > > > > > > > > >
> > > > > > > > > > > > The act of 'choosing' a mirror could be done in a
> > number
> > > of
> > > > > > ways.
> > > > > > > > > > > > - If we want to define an order, then it could just try
> > > > from
> > > > > > the
> > > > > > > > top
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > list and work its way down.  It would curl the target
> > to
> > > > make
> > > > > > > sure
> > > > > > > > it
> > > > > > > > > > > gets
> > > > > > > > > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > > > > > > > > - Or, if we want to distribute the load across the
> > > mirrors,
> > > > > we
> > > > > > > > could
> > > > > > > > > > pick
> > > > > > > > > > > > from the list randomly.  Again, doing a curl to verify
> > > the
> > > > > > mirror
> > > > > > > > is
> > > > > > > > > up
> > > > > > > > > > > and
> > > > > > > > > > > > then doing a redirect.
> > > > > > > > > > > > - If we want to get fancy, we could do a reverse IP
> > > lookup
> > > > > and
> > > > > > > try
> > > > > > > > to
> > > > > > > > > > > match
> > > > > > > > > > > > the requester with their closest geographical mirror.
> > > > > > > > > > > >
> > > > > > > > > > > > Thoughts?
> > > > > > > > > > > >
> > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > Lead Developer
> > > > > > > > > > > >
> > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > > > > > > > chiradeepv@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > What steps are needed to set up a mirror? What does
> > > Infra
> > > > > > need
> > > > > > > to
> > > > > > > > > do?
> > > > > > > > > > > Has
> > > > > > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > > > > > > raja.pullela@accelerite.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi will,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I believe, we didn’t get to close ‘getting a mirror
> > > on
> > > > > > > Apache’
> > > > > > > > > > > because
> > > > > > > > > > > > we
> > > > > > > > > > > > > > needed someone on the Apache Infra side to close
> > > this.
> > > > > BTW,
> > > > > > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages this?)
> > > has
> > > > > > > all/most
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > > content.  Once we can close on the Apache mirror
> > for
> > > > > > hosting
> > > > > > > > the
> > > > > > > > > > > > > content, I
> > > > > > > > > > > > > > can help assist getting the content there.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For now, we have replicated the download.cloud.com
> > > > > content
> > > > > > > to
> > > > > > > > ‘
> > > > > > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > > > > > Also, we are working on a set of steps/procedure to
> > > > help
> > > > > > with
> > > > > > > > > this
> > > > > > > > > > > > > > change.  I will update everyone in about a week’s
> > > time
> > > > on
> > > > > > the
> > > > > > > > > > > details.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Raja Pullela
> > > > > > > > > > > > > > Engineering Team,
> > > > > > > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on
> > > > > behalf
> > > > > > of
> > > > > > > > > Will
> > > > > > > > > > > > > > Stevens" <williamstevens@gmail.com on behalf of
> > > > > > > > > > > wstevens@cloudops.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > unfortunately the template mirror conversation got
> > > > caught
> > > > > > up
> > > > > > > in
> > > > > > > > > > > details
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > nobody took the lead on implementing a solution.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > citrix has been pinging me every couple months to
> > say
> > > > > > 'dude,
> > > > > > > we
> > > > > > > > > > need
> > > > > > > > > > > to
> > > > > > > > > > > > > > remove the dependency on download.citrix.com',
> > but i
> > > > > have
> > > > > > > not
> > > > > > > > > had
> > > > > > > > > > > the
> > > > > > > > > > > > > > cycles to get in and solve the problem.  the
> > shutdown
> > > > of
> > > > > > that
> > > > > > > > is
> > > > > > > > > > > > imminent
> > > > > > > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Nathan,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Ideally, if you put the template location in (or
> > > use
> > > > a
> > > > > > > > template
> > > > > > > > > > > > defined
> > > > > > > > > > > > > > > in)  test_data.py then the actual location can be
> > > > > > > overridden
> > > > > > > > by
> > > > > > > > > > > > anyone
> > > > > > > > > > > > > > > testing.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For Trillian, we've copied all of the templates
> > > that
> > > > > > people
> > > > > > > > > have
> > > > > > > > > > > > define
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > a local repo and then replace the URLs in
> > > > test_data.py
> > > > > to
> > > > > > > > > reduce
> > > > > > > > > > > > > > bandwidth
> > > > > > > > > > > > > > > use and download times.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Ie:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >             "bootableIso":
> > > > > > > > > > > > > > >                 {
> > > > > > > > > > > > > > >                     "displaytext": "Test Bootable
> > > > ISO",
> > > > > > > > > > > > > > >                     "name": "testISO",
> > > > > > > > > > > > > > >                     "bootable": True,
> > > > > > > > > > > > > > >                     "ispublic": False,
> > > > > > > > > > > > > > >                     "url": "{{
> > > marvin_images_location
> > > > > > > > > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > > > > > > > >                     "ostype": 'Other Linux
> > > (64-bit)',
> > > > > > > > > > > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > > > > > > > > > > >         },
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I thought that we had come up with a solution for
> > > > > > > > > > > download.cloud.com
> > > > > > > > > > > > ,
> > > > > > > > > > > > > by
> > > > > > > > > > > > > > > having a mirrorlist hosted in Community Apache
> > > > 'space'
> > > > > > with
> > > > > > > > > > anyone
> > > > > > > > > > > > able
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > out themselves forward as a mirror.
> > > > > > > > > > > > > > > But I must admit I lost track of whether anyone
> > > made
> > > > > the
> > > > > > > > > > requisite
> > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > in code....
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Kind regards,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Paul Angus
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > > > > > www.shapeblue.com
> > > > > > > > > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N
> > 4HSUK
> > > > > > > > > > > > > > > @shapeblue
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > > > > From: williamstevens@gmail.com [mailto:
> > > > > > > williamstevens@gmail.
> > > > > > > > > com]
> > > > > > > > > > > On
> > > > > > > > > > > > > > > Behalf Of Will Stevens
> > > > > > > > > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > > > > > > > Subject: Re: Modern template hosting
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > this is a hard questions.  in general, we should
> > be
> > > > > > setting
> > > > > > > > up
> > > > > > > > > a
> > > > > > > > > > > > mirror
> > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > some cloudstack/apache domain and then mirror to
> > > > other
> > > > > > > > provided
> > > > > > > > > > > > > > templates.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > we MUST come up with a solution to deprecate '
> > > > > > > > > download.cloud.com
> > > > > > > > > > ',
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > is going to be going away any day now.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > i don't know the right way to solve this to be
> > > > honest,
> > > > > > but
> > > > > > > if
> > > > > > > > > you
> > > > > > > > > > > > have
> > > > > > > > > > > > > > > ideas, i am willing to help.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson
> > <
> > > > > > > > > > njohnson@ena.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > So not to re-open a can of worms, but I’m in a
> > > > > > situation
> > > > > > > > > where
> > > > > > > > > > I
> > > > > > > > > > > > need
> > > > > > > > > > > > > > > > to come up with a Marvin component test that
> > > > depends
> > > > > > on a
> > > > > > > > > > > template
> > > > > > > > > > > > > > > > based on a kernel that’s relatively new, i.e.,
> > > > newer
> > > > > > than
> > > > > > > > > > Centos
> > > > > > > > > > > > 5.3
> > > > > > > > > > > > > /
> > > > > > > > > > > > > > > Ubuntu 10.04 .
> > > > > > > > > > > > > > > > I see openvm.eu has a suitable template
> > (Ubuntu
> > > > > 16.0.4
> > > > > > > for
> > > > > > > > > > KVM),
> > > > > > > > > > > > but
> > > > > > > > > > > > > > > > from looking at the thread "Migrating
> > CloudStack
> > > > > > content
> > > > > > > > from
> > > > > > > > > > > > > > > > download.cloud.com” it looks like there is
> > > > > resistance
> > > > > > to
> > > > > > > > > using
> > > > > > > > > > > > this
> > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > least for hosting system vm templates over
> > > concerns
> > > > > of
> > > > > > > > > > > neutrality.
> > > > > > > > > > > > > > > > Would this be suitable for a component test?
> > If
> > > > not,
> > > > > > > what
> > > > > > > > > is a
> > > > > > > > > > > > > > “blessed”
> > > > > > > > > > > > > > > > template location?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks in advance!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Nathan Johnson
> > > > > > > > > > > > > > > > R&D Engineer
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > > > > > > > > Nashville, TN 37211
> > > > > > > > > > > > > > > > General Office: 615-312-6000
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > website | blog | support
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > DISCLAIMER
> > > > > > > > > > > > > > ==========
> > > > > > > > > > > > > > This e-mail may contain privileged and confidential
> > > > > > > information
> > > > > > > > > > which
> > > > > > > > > > > > is
> > > > > > > > > > > > > > the property of Accelerite, a Persistent Systems
> > > > > business.
> > > > > > It
> > > > > > > > is
> > > > > > > > > > > > intended
> > > > > > > > > > > > > > only for the use of the individual or entity to
> > which
> > > > it
> > > > > is
> > > > > > > > > > > addressed.
> > > > > > > > > > > > If
> > > > > > > > > > > > > > you are not the intended recipient, you are not
> > > > > authorized
> > > > > > to
> > > > > > > > > read,
> > > > > > > > > > > > > retain,
> > > > > > > > > > > > > > copy, print, distribute or use this message. If you
> > > > have
> > > > > > > > received
> > > > > > > > > > > this
> > > > > > > > > > > > > > communication in error, please notify the sender
> > and
> > > > > delete
> > > > > > > all
> > > > > > > > > > > copies
> > > > > > > > > > > > of
> > > > > > > > > > > > > > this message. Accelerite, a Persistent Systems
> > > business
> > > > > > does
> > > > > > > > not
> > > > > > > > > > > accept
> > > > > > > > > > > > > any
> > > > > > > > > > > > > > liability for virus infected mails.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Rafael Weingärtner
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rafael Weingärtner
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> > >
> >

Re: Modern template hosting

Posted by Rajani Karuturi <ra...@apache.org>.

Pierre-Luc did an amazing job in getting
https://builds.cloudstack.org/ working [1].

We can work on similar lines and probably get
packages.cloudstack.org working with mirrored entries form
apt-get.eu or openvm.eu or s3.downloads.accelerite or
packages.shapeblue etc.

[1] https://issues.apache.org/jira/browse/INFRA-11903

Thanks,

~ Rajani

http://cloudplatform.accelerite.com/

On February 28, 2017 at 2:13 PM, Erik Weber (terbolous@gmail.com)
wrote:

On Tue, Feb 28, 2017 at 8:59 AM, Daan Hoogland
<da...@gmail.com> wrote:

On Tue, Feb 28, 2017 at 2:40 AM, Pierre-Luc Dion
<pd...@apache.org> wrote:

For our systemvm templates, can we just change the URL for
somthing like
cloudstack.apache.org/systemvm/... ?

I agree,
is there any change we can host a domain for this at apache, if
we
can't keep downloads.cloud.com then maybe
downloads.cloudstack.com. To
many people are used to this this way of working and a patch for
a
name change is the easiest. Devising a new mechanism is nice but
a
different job, is it?

packages.apache.org exists.

Has been discussed before in a thread called 'Package
Repositories'.

One thing that should be focused on is that if we go the ASF
hosted
route, we should be able to update the content without involving
ASF
Infra team.

--
Erik

Re: Modern template hosting

Posted by Erik Weber <te...@gmail.com>.

On Tue, Feb 28, 2017 at 8:59 AM, Daan Hoogland <da...@gmail.com> wrote:
> On Tue, Feb 28, 2017 at 2:40 AM, Pierre-Luc Dion <pd...@apache.org> wrote:
>> For our systemvm templates, can we just change the URL for somthing like
>> cloudstack.apache.org/systemvm/... ?
>
> I agree,
> is there any change we can host a domain for this at apache, if we
> can't keep downloads.cloud.com then maybe downloads.cloudstack.com. To
> many people are used to this this way of working and a patch for a
> name change is the easiest. Devising a new mechanism is nice but a
> different job, is it?

packages.apache.org exists.

Has been discussed before in a thread called 'Package Repositories'.

One thing that should be focused on is that if we go the ASF hosted
route, we should be able to update the content without involving ASF
Infra team.

-- 
Erik

Re: Modern template hosting

Posted by Daan Hoogland <da...@gmail.com>.

On Tue, Feb 28, 2017 at 2:40 AM, Pierre-Luc Dion <pd...@apache.org> wrote:
> For our systemvm templates, can we just change the URL for somthing like
> cloudstack.apache.org/systemvm/... ?

I agree,
is there any change we can host a domain for this at apache, if we
can't keep downloads.cloud.com then maybe downloads.cloudstack.com. To
many people are used to this this way of working and a patch for a
name change is the easiest. Devising a new mechanism is nice but a
different job, is it?

-- 
Daan

Re: Modern template hosting

Posted by Pierre-Luc Dion <pd...@apache.org>.

hi!

I think we should work with distro provider to have their cloud builds work
with cloudstack. Good example is CoreOS, it work out of the box from their
channel builds.
it shouldn't be too complicated to have centos, ubuntu and debian, unless
...

For our systemvm templates, can we just change the URL for somthing like
cloudstack.apache.org/systemvm/... ?
and for old systemvm that are depricated but that we might want to keep,
could we archive them into a github repo in
https://github.com/apachecloudstack ?




On Mon, Feb 27, 2017 at 6:28 PM, Chiradeep Vittal <ch...@gmail.com>
wrote:

> My stance is that the current workflow does a disservice to the user
> community by letting them install / use outdated and insecure templates.
> Now, let's assume download.cloud.com is gone forever. What do we tell ACS
> users pre-4.11 as far as *built-in templates* go?
> 1. Direct them to update templates.sql with some new URL, but with the same
> dirty old templates
> 2. Direct them to update templates.sql with some new URL, but with nice
> templates (e.g., open.vm.eu)
> 3. Same as (2), but document more choices.
>
> Now, why should things be different for 4.11 and later? Documenting the
> steps to install templates offline is trivial (and can be scripted to a
> large part, like cloud-install-sys-tmplt)
>
> For pre-4.11 users, for *systemvms*, anyway we tell them to use
> http://cloudstack.apt-get.eu which is not controlled by ACS.
>
>
> On Mon, Feb 27, 2017 at 2:50 PM, Rafael Weingärtner <
> rafaelweingartner@gmail.com> wrote:
>
> > Agree with you.
> > We need to support the current working flow. And then, define the first
> > version that will start using the new approach.
> >
> > On Mon, Feb 27, 2017 at 5:36 PM, Will Stevens <ws...@cloudops.com>
> > wrote:
> >
> > > I think we almost need a two pronged approach.
> > >
> > > 1) Get a solution in place which will enable us to document and serve
> > > templates for legacy systems.  I will work on this.
> > > 2) Discuss and understand how we SHOULD be handling this problem in the
> > > future and in what release we can expect it.
> > >
> > > I think we need to do both.  I think we should start to try to really
> > > understand what we want to deliver in (2) going forward.
> > >
> > > *Will STEVENS*
> > > Lead Developer
> > >
> > > <https://goo.gl/NYZ8KK>
> > >
> > > On Mon, Feb 27, 2017 at 4:53 PM, Rafael Weingärtner <
> > > rafaelweingartner@gmail.com> wrote:
> > >
> > > > My worry is exactly with system VMs templates.
> > > >
> > > > Currently, we indicate administrators to download them from
> > > > http://cloudstack.apt-get.eu/systemvm/4.6/ [1]. However, the
> > > installation
> > > > docs do not mention the expected hashes for the file that is going to
> > be
> > > > downloaded.
> > > > Also, I do not know the code that downloads system VMs templates
> (when
> > > > upgrading), but if the hash being checked is taken from the mirror
> used
> > > to
> > > > download the file; the only thing it checks is that if the download
> > > > finished successfully (no transmission errors). If we want to check
> > > > integrity, check that the template we created is untampered; we need
> to
> > > > host and serve the hash in a secure manner.
> > > >
> > > > [1]
> > > > http://docs.cloudstack.apache.org/projects/cloudstack-
> > > installation/en/4.9/
> > > > management-server/index.html#prepare-the-system-vm-template
> > > >
> > > >
> > > > On Mon, Feb 27, 2017 at 4:36 PM, Chiradeep Vittal <
> > chiradeepv@gmail.com>
> > > > wrote:
> > > >
> > > > > Hashes are checked (md5 IIRC) today.
> > > > > But given the issues, I think the project should steer away from
> > > hosting
> > > > > templates except the systemvm template.
> > > > >
> > > > > On Mon, Feb 27, 2017 at 1:31 PM, Rafael Weingärtner <
> > > > > rafaelweingartner@gmail.com> wrote:
> > > > >
> > > > > > Will, I think we could support different path structures. This
> can
> > > > > > facilitate different deployment of mirrors based on the structure
> > the
> > > > > host
> > > > > > has.
> > > > > >
> > > > > > Could I add something else to the discussion? Have we discussed
> the
> > > > > > security impacts of setting up this mirrors approach?
> > > > > > I mean, if any of the mirrors gets corrupted (let`s say by a
> > hacker),
> > > > and
> > > > > > the templates are injected with malicious code, an attacker could
> > > > > > potentially get un-monitored and unlimited access to a cloud
> > > > environment.
> > > > > >
> > > > > > If we assume that the mirror may get malicious (it is not that I
> do
> > > not
> > > > > > trust you guys, but bad things happen), we cannot host hashes
> > there.
> > > > > Where
> > > > > > do you think we could store Sha512 or another hash type for these
> > > > > > templates? Could we host in the newly proposed Github repo or
> maybe
> > > > some
> > > > > > place in the ACS website?
> > > > > >
> > > > > > This would have an impact on clients (needing clear
> documentation)
> > > and
> > > > > our
> > > > > > code that automatically downloads System VM templates (does it
> > check
> > > > > hashes
> > > > > > when automatically installing templates today? It may require
> > > > > > implementation changes).
> > > > > >
> > > > > > On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <
> > wstevens@cloudops.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > so this is what I am looking to do.  Please let me know if you
> > have
> > > > > > > suggestions for me or think I should be solving the problem a
> > > > different
> > > > > > > way.
> > > > > > >
> > > > > > > - We request a new Github repository from the ASF at:
> > > > > > > 'apache/cloudstack-mirror-list'
> > > > > > > - In this repository we track a text file in the 'gh-pages'
> > branch
> > > > > with a
> > > > > > > list of valid download mirrors.
> > > > > > > - I build a binary to be hosted by the ASF (or at least with
> the
> > > ASF
> > > > > > > pointing a domain at the binary and I could potentially host
> it).
> > > We
> > > > > > will
> > > > > > > see how they want to handle the hosting of the binary.
> > > > > > >
> > > > > > > The binary would expose a web server which would behave as
> > follows:
> > > > > > > - When the 'client' requests a download url the following flow
> is
> > > > > kicked
> > > > > > > off:
> > > > > > > -- The mirror list is queried from github (or from a static
> site
> > > > hosted
> > > > > > on
> > > > > > > asf, as we see fit).
> > > > > > > -- The Lat/Lon of the 'client' is determined based on their IP.
> > > > > > > -- The Lat/Lon for each of the 'mirror's is determined based on
> > an
> > > IP
> > > > > > > lookup of the hostname.
> > > > > > > -- The closest geographical mirror is determined, the target is
> > > > > validated
> > > > > > > to be available and the user is redirected.
> > > > > > >
> > > > > > > Some questions I have right now:
> > > > > > > - Will every mirror have the same path structure to access the
> > > > > equivalent
> > > > > > > resources?
> > > > > > > - Should we support adding a path to the mirror url to specify
> > the
> > > > path
> > > > > > to
> > > > > > > the base common path?
> > > > > > > -- Example: lets say the binary is hosted on 'dl.acs.com' and
> > > there
> > > > > are
> > > > > > > three mirrors 'abc.com', 'pqr.com/files' and '
> xyx.com/downloads
> > '.
> > > > > > > -- If the path being requested is '
> > > > > > > dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would
> result
> > in
> > > > the
> > > > > > > following potential paths for the mirrors:
> > > > > > > -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > > -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > > -- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > >
> > > > > > > Does this all make sense?
> > > > > > >
> > > > > > > *Will STEVENS*
> > > > > > > Lead Developer
> > > > > > >
> > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > >
> > > > > > > On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <
> > > > > chiradeepv@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > My bad. A few lines down, this has been added recently:
> > > > > > > >
> > > > > > > > this.request.setFollowRedirects(true);
> > > > > > > >
> > > > > > > > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <
> > > > > > williamstevens@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > OK. Thanks for the heads up.
> > > > > > > > >
> > > > > > > > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > chiradeepv@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Sounds workable. The downloader code in the SSVM won't
> > follow
> > > > > > > > redirects I
> > > > > > > > > > think.
> > > > > > > > > > https://github.com/apache/cloudstack/blob/
> > > > > > > > 5511065fc20787619d9cd0444a65a3
> > > > > > > > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > > > > > > > HttpTemplateDownloader.java#L93
> > > > > > > > > > https://goo.gl/dSi0r5
> > > > > > > > > >  Might need to add
> > > > > > > > > > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <
> > > > > > wstevens@cloudops.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > We haven't opened a ticket yet because we don't have a
> > > > strategy
> > > > > > > yet.
> > > > > > > > > > >
> > > > > > > > > > > What do you guys think of this:
> > > > > > > > > > > - We setup a new github repo in the 'apache' org which
> > > > consists
> > > > > > of
> > > > > > > a
> > > > > > > > > > single
> > > > > > > > > > > file with a list of active/supported mirrors.
> > > > > > > > > > > - I write a small web server, distributed as a binary,
> > > which
> > > > > can
> > > > > > be
> > > > > > > > > > hosted
> > > > > > > > > > > by ASF Infra.  This web server will query the current
> > list
> > > of
> > > > > > > mirrors
> > > > > > > > > and
> > > > > > > > > > > will select one and then do a 302 redirect to that
> > mirror.
> > > > > > > > > > >
> > > > > > > > > > > The act of 'choosing' a mirror could be done in a
> number
> > of
> > > > > ways.
> > > > > > > > > > > - If we want to define an order, then it could just try
> > > from
> > > > > the
> > > > > > > top
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > list and work its way down.  It would curl the target
> to
> > > make
> > > > > > sure
> > > > > > > it
> > > > > > > > > > gets
> > > > > > > > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > > > > > > > - Or, if we want to distribute the load across the
> > mirrors,
> > > > we
> > > > > > > could
> > > > > > > > > pick
> > > > > > > > > > > from the list randomly.  Again, doing a curl to verify
> > the
> > > > > mirror
> > > > > > > is
> > > > > > > > up
> > > > > > > > > > and
> > > > > > > > > > > then doing a redirect.
> > > > > > > > > > > - If we want to get fancy, we could do a reverse IP
> > lookup
> > > > and
> > > > > > try
> > > > > > > to
> > > > > > > > > > match
> > > > > > > > > > > the requester with their closest geographical mirror.
> > > > > > > > > > >
> > > > > > > > > > > Thoughts?
> > > > > > > > > > >
> > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > Lead Developer
> > > > > > > > > > >
> > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > > > > > > chiradeepv@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > What steps are needed to set up a mirror? What does
> > Infra
> > > > > need
> > > > > > to
> > > > > > > > do?
> > > > > > > > > > Has
> > > > > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > > > > > raja.pullela@accelerite.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi will,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I believe, we didn’t get to close ‘getting a mirror
> > on
> > > > > > Apache’
> > > > > > > > > > because
> > > > > > > > > > > we
> > > > > > > > > > > > > needed someone on the Apache Infra side to close
> > this.
> > > > BTW,
> > > > > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages this?)
> > has
> > > > > > all/most
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > content.  Once we can close on the Apache mirror
> for
> > > > > hosting
> > > > > > > the
> > > > > > > > > > > > content, I
> > > > > > > > > > > > > can help assist getting the content there.
> > > > > > > > > > > > >
> > > > > > > > > > > > > For now, we have replicated the download.cloud.com
> > > > content
> > > > > > to
> > > > > > > ‘
> > > > > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > > > > Also, we are working on a set of steps/procedure to
> > > help
> > > > > with
> > > > > > > > this
> > > > > > > > > > > > > change.  I will update everyone in about a week’s
> > time
> > > on
> > > > > the
> > > > > > > > > > details.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Raja Pullela
> > > > > > > > > > > > > Engineering Team,
> > > > > > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on
> > > > behalf
> > > > > of
> > > > > > > > Will
> > > > > > > > > > > > > Stevens" <williamstevens@gmail.com on behalf of
> > > > > > > > > > wstevens@cloudops.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > unfortunately the template mirror conversation got
> > > caught
> > > > > up
> > > > > > in
> > > > > > > > > > details
> > > > > > > > > > > > and
> > > > > > > > > > > > > nobody took the lead on implementing a solution.
> > > > > > > > > > > > >
> > > > > > > > > > > > > citrix has been pinging me every couple months to
> say
> > > > > 'dude,
> > > > > > we
> > > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > > > > remove the dependency on download.citrix.com',
> but i
> > > > have
> > > > > > not
> > > > > > > > had
> > > > > > > > > > the
> > > > > > > > > > > > > cycles to get in and solve the problem.  the
> shutdown
> > > of
> > > > > that
> > > > > > > is
> > > > > > > > > > > imminent
> > > > > > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > >
> > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Nathan,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Ideally, if you put the template location in (or
> > use
> > > a
> > > > > > > template
> > > > > > > > > > > defined
> > > > > > > > > > > > > > in)  test_data.py then the actual location can be
> > > > > > overridden
> > > > > > > by
> > > > > > > > > > > anyone
> > > > > > > > > > > > > > testing.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For Trillian, we've copied all of the templates
> > that
> > > > > people
> > > > > > > > have
> > > > > > > > > > > define
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > a local repo and then replace the URLs in
> > > test_data.py
> > > > to
> > > > > > > > reduce
> > > > > > > > > > > > > bandwidth
> > > > > > > > > > > > > > use and download times.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Ie:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >             "bootableIso":
> > > > > > > > > > > > > >                 {
> > > > > > > > > > > > > >                     "displaytext": "Test Bootable
> > > ISO",
> > > > > > > > > > > > > >                     "name": "testISO",
> > > > > > > > > > > > > >                     "bootable": True,
> > > > > > > > > > > > > >                     "ispublic": False,
> > > > > > > > > > > > > >                     "url": "{{
> > marvin_images_location
> > > > > > > > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > > > > > > >                     "ostype": 'Other Linux
> > (64-bit)',
> > > > > > > > > > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > > > > > > > > > >         },
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I thought that we had come up with a solution for
> > > > > > > > > > download.cloud.com
> > > > > > > > > > > ,
> > > > > > > > > > > > by
> > > > > > > > > > > > > > having a mirrorlist hosted in Community Apache
> > > 'space'
> > > > > with
> > > > > > > > > anyone
> > > > > > > > > > > able
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > out themselves forward as a mirror.
> > > > > > > > > > > > > > But I must admit I lost track of whether anyone
> > made
> > > > the
> > > > > > > > > requisite
> > > > > > > > > > > > > changes
> > > > > > > > > > > > > > in code....
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Kind regards,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Paul Angus
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > > > > www.shapeblue.com
> > > > > > > > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N
> 4HSUK
> > > > > > > > > > > > > > @shapeblue
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > > > From: williamstevens@gmail.com [mailto:
> > > > > > williamstevens@gmail.
> > > > > > > > com]
> > > > > > > > > > On
> > > > > > > > > > > > > > Behalf Of Will Stevens
> > > > > > > > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > > > > > > Subject: Re: Modern template hosting
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > this is a hard questions.  in general, we should
> be
> > > > > setting
> > > > > > > up
> > > > > > > > a
> > > > > > > > > > > mirror
> > > > > > > > > > > > > on
> > > > > > > > > > > > > > some cloudstack/apache domain and then mirror to
> > > other
> > > > > > > provided
> > > > > > > > > > > > > templates.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > we MUST come up with a solution to deprecate '
> > > > > > > > download.cloud.com
> > > > > > > > > ',
> > > > > > > > > > > > that
> > > > > > > > > > > > > > is going to be going away any day now.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > i don't know the right way to solve this to be
> > > honest,
> > > > > but
> > > > > > if
> > > > > > > > you
> > > > > > > > > > > have
> > > > > > > > > > > > > > ideas, i am willing to help.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson
> <
> > > > > > > > > njohnson@ena.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > So not to re-open a can of worms, but I’m in a
> > > > > situation
> > > > > > > > where
> > > > > > > > > I
> > > > > > > > > > > need
> > > > > > > > > > > > > > > to come up with a Marvin component test that
> > > depends
> > > > > on a
> > > > > > > > > > template
> > > > > > > > > > > > > > > based on a kernel that’s relatively new, i.e.,
> > > newer
> > > > > than
> > > > > > > > > Centos
> > > > > > > > > > > 5.3
> > > > > > > > > > > > /
> > > > > > > > > > > > > > Ubuntu 10.04 .
> > > > > > > > > > > > > > > I see openvm.eu has a suitable template
> (Ubuntu
> > > > 16.0.4
> > > > > > for
> > > > > > > > > KVM),
> > > > > > > > > > > but
> > > > > > > > > > > > > > > from looking at the thread "Migrating
> CloudStack
> > > > > content
> > > > > > > from
> > > > > > > > > > > > > > > download.cloud.com” it looks like there is
> > > > resistance
> > > > > to
> > > > > > > > using
> > > > > > > > > > > this
> > > > > > > > > > > > at
> > > > > > > > > > > > > > > least for hosting system vm templates over
> > concerns
> > > > of
> > > > > > > > > > neutrality.
> > > > > > > > > > > > > > > Would this be suitable for a component test?
> If
> > > not,
> > > > > > what
> > > > > > > > is a
> > > > > > > > > > > > > “blessed”
> > > > > > > > > > > > > > > template location?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks in advance!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Nathan Johnson
> > > > > > > > > > > > > > > R&D Engineer
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > > > > > > > Nashville, TN 37211
> > > > > > > > > > > > > > > General Office: 615-312-6000
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > website | blog | support
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > DISCLAIMER
> > > > > > > > > > > > > ==========
> > > > > > > > > > > > > This e-mail may contain privileged and confidential
> > > > > > information
> > > > > > > > > which
> > > > > > > > > > > is
> > > > > > > > > > > > > the property of Accelerite, a Persistent Systems
> > > > business.
> > > > > It
> > > > > > > is
> > > > > > > > > > > intended
> > > > > > > > > > > > > only for the use of the individual or entity to
> which
> > > it
> > > > is
> > > > > > > > > > addressed.
> > > > > > > > > > > If
> > > > > > > > > > > > > you are not the intended recipient, you are not
> > > > authorized
> > > > > to
> > > > > > > > read,
> > > > > > > > > > > > retain,
> > > > > > > > > > > > > copy, print, distribute or use this message. If you
> > > have
> > > > > > > received
> > > > > > > > > > this
> > > > > > > > > > > > > communication in error, please notify the sender
> and
> > > > delete
> > > > > > all
> > > > > > > > > > copies
> > > > > > > > > > > of
> > > > > > > > > > > > > this message. Accelerite, a Persistent Systems
> > business
> > > > > does
> > > > > > > not
> > > > > > > > > > accept
> > > > > > > > > > > > any
> > > > > > > > > > > > > liability for virus infected mails.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Rafael Weingärtner
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Rafael Weingärtner
> > > >
> > >
> >
> >
> >
> > --
> > Rafael Weingärtner
> >
>

Re: Modern template hosting

Posted by Chiradeep Vittal <ch...@gmail.com>.

My stance is that the current workflow does a disservice to the user
community by letting them install / use outdated and insecure templates.
Now, let's assume download.cloud.com is gone forever. What do we tell ACS
users pre-4.11 as far as *built-in templates* go?
1. Direct them to update templates.sql with some new URL, but with the same
dirty old templates
2. Direct them to update templates.sql with some new URL, but with nice
templates (e.g., open.vm.eu)
3. Same as (2), but document more choices.

Now, why should things be different for 4.11 and later? Documenting the
steps to install templates offline is trivial (and can be scripted to a
large part, like cloud-install-sys-tmplt)

For pre-4.11 users, for *systemvms*, anyway we tell them to use
http://cloudstack.apt-get.eu which is not controlled by ACS.


On Mon, Feb 27, 2017 at 2:50 PM, Rafael Weingärtner <
rafaelweingartner@gmail.com> wrote:

> Agree with you.
> We need to support the current working flow. And then, define the first
> version that will start using the new approach.
>
> On Mon, Feb 27, 2017 at 5:36 PM, Will Stevens <ws...@cloudops.com>
> wrote:
>
> > I think we almost need a two pronged approach.
> >
> > 1) Get a solution in place which will enable us to document and serve
> > templates for legacy systems.  I will work on this.
> > 2) Discuss and understand how we SHOULD be handling this problem in the
> > future and in what release we can expect it.
> >
> > I think we need to do both.  I think we should start to try to really
> > understand what we want to deliver in (2) going forward.
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Mon, Feb 27, 2017 at 4:53 PM, Rafael Weingärtner <
> > rafaelweingartner@gmail.com> wrote:
> >
> > > My worry is exactly with system VMs templates.
> > >
> > > Currently, we indicate administrators to download them from
> > > http://cloudstack.apt-get.eu/systemvm/4.6/ [1]. However, the
> > installation
> > > docs do not mention the expected hashes for the file that is going to
> be
> > > downloaded.
> > > Also, I do not know the code that downloads system VMs templates (when
> > > upgrading), but if the hash being checked is taken from the mirror used
> > to
> > > download the file; the only thing it checks is that if the download
> > > finished successfully (no transmission errors). If we want to check
> > > integrity, check that the template we created is untampered; we need to
> > > host and serve the hash in a secure manner.
> > >
> > > [1]
> > > http://docs.cloudstack.apache.org/projects/cloudstack-
> > installation/en/4.9/
> > > management-server/index.html#prepare-the-system-vm-template
> > >
> > >
> > > On Mon, Feb 27, 2017 at 4:36 PM, Chiradeep Vittal <
> chiradeepv@gmail.com>
> > > wrote:
> > >
> > > > Hashes are checked (md5 IIRC) today.
> > > > But given the issues, I think the project should steer away from
> > hosting
> > > > templates except the systemvm template.
> > > >
> > > > On Mon, Feb 27, 2017 at 1:31 PM, Rafael Weingärtner <
> > > > rafaelweingartner@gmail.com> wrote:
> > > >
> > > > > Will, I think we could support different path structures. This can
> > > > > facilitate different deployment of mirrors based on the structure
> the
> > > > host
> > > > > has.
> > > > >
> > > > > Could I add something else to the discussion? Have we discussed the
> > > > > security impacts of setting up this mirrors approach?
> > > > > I mean, if any of the mirrors gets corrupted (let`s say by a
> hacker),
> > > and
> > > > > the templates are injected with malicious code, an attacker could
> > > > > potentially get un-monitored and unlimited access to a cloud
> > > environment.
> > > > >
> > > > > If we assume that the mirror may get malicious (it is not that I do
> > not
> > > > > trust you guys, but bad things happen), we cannot host hashes
> there.
> > > > Where
> > > > > do you think we could store Sha512 or another hash type for these
> > > > > templates? Could we host in the newly proposed Github repo or maybe
> > > some
> > > > > place in the ACS website?
> > > > >
> > > > > This would have an impact on clients (needing clear documentation)
> > and
> > > > our
> > > > > code that automatically downloads System VM templates (does it
> check
> > > > hashes
> > > > > when automatically installing templates today? It may require
> > > > > implementation changes).
> > > > >
> > > > > On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <
> wstevens@cloudops.com
> > >
> > > > > wrote:
> > > > >
> > > > > > so this is what I am looking to do.  Please let me know if you
> have
> > > > > > suggestions for me or think I should be solving the problem a
> > > different
> > > > > > way.
> > > > > >
> > > > > > - We request a new Github repository from the ASF at:
> > > > > > 'apache/cloudstack-mirror-list'
> > > > > > - In this repository we track a text file in the 'gh-pages'
> branch
> > > > with a
> > > > > > list of valid download mirrors.
> > > > > > - I build a binary to be hosted by the ASF (or at least with the
> > ASF
> > > > > > pointing a domain at the binary and I could potentially host it).
> > We
> > > > > will
> > > > > > see how they want to handle the hosting of the binary.
> > > > > >
> > > > > > The binary would expose a web server which would behave as
> follows:
> > > > > > - When the 'client' requests a download url the following flow is
> > > > kicked
> > > > > > off:
> > > > > > -- The mirror list is queried from github (or from a static site
> > > hosted
> > > > > on
> > > > > > asf, as we see fit).
> > > > > > -- The Lat/Lon of the 'client' is determined based on their IP.
> > > > > > -- The Lat/Lon for each of the 'mirror's is determined based on
> an
> > IP
> > > > > > lookup of the hostname.
> > > > > > -- The closest geographical mirror is determined, the target is
> > > > validated
> > > > > > to be available and the user is redirected.
> > > > > >
> > > > > > Some questions I have right now:
> > > > > > - Will every mirror have the same path structure to access the
> > > > equivalent
> > > > > > resources?
> > > > > > - Should we support adding a path to the mirror url to specify
> the
> > > path
> > > > > to
> > > > > > the base common path?
> > > > > > -- Example: lets say the binary is hosted on 'dl.acs.com' and
> > there
> > > > are
> > > > > > three mirrors 'abc.com', 'pqr.com/files' and 'xyx.com/downloads
> '.
> > > > > > -- If the path being requested is '
> > > > > > dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would result
> in
> > > the
> > > > > > following potential paths for the mirrors:
> > > > > > -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > -- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > >
> > > > > > Does this all make sense?
> > > > > >
> > > > > > *Will STEVENS*
> > > > > > Lead Developer
> > > > > >
> > > > > > <https://goo.gl/NYZ8KK>
> > > > > >
> > > > > > On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <
> > > > chiradeepv@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > My bad. A few lines down, this has been added recently:
> > > > > > >
> > > > > > > this.request.setFollowRedirects(true);
> > > > > > >
> > > > > > > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <
> > > > > williamstevens@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > OK. Thanks for the heads up.
> > > > > > > >
> > > > > > > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > chiradeepv@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Sounds workable. The downloader code in the SSVM won't
> follow
> > > > > > > redirects I
> > > > > > > > > think.
> > > > > > > > > https://github.com/apache/cloudstack/blob/
> > > > > > > 5511065fc20787619d9cd0444a65a3
> > > > > > > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > > > > > > HttpTemplateDownloader.java#L93
> > > > > > > > > https://goo.gl/dSi0r5
> > > > > > > > >  Might need to add
> > > > > > > > > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > > > > > > >
> > > > > > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <
> > > > > wstevens@cloudops.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > We haven't opened a ticket yet because we don't have a
> > > strategy
> > > > > > yet.
> > > > > > > > > >
> > > > > > > > > > What do you guys think of this:
> > > > > > > > > > - We setup a new github repo in the 'apache' org which
> > > consists
> > > > > of
> > > > > > a
> > > > > > > > > single
> > > > > > > > > > file with a list of active/supported mirrors.
> > > > > > > > > > - I write a small web server, distributed as a binary,
> > which
> > > > can
> > > > > be
> > > > > > > > > hosted
> > > > > > > > > > by ASF Infra.  This web server will query the current
> list
> > of
> > > > > > mirrors
> > > > > > > > and
> > > > > > > > > > will select one and then do a 302 redirect to that
> mirror.
> > > > > > > > > >
> > > > > > > > > > The act of 'choosing' a mirror could be done in a number
> of
> > > > ways.
> > > > > > > > > > - If we want to define an order, then it could just try
> > from
> > > > the
> > > > > > top
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > list and work its way down.  It would curl the target to
> > make
> > > > > sure
> > > > > > it
> > > > > > > > > gets
> > > > > > > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > > > > > > - Or, if we want to distribute the load across the
> mirrors,
> > > we
> > > > > > could
> > > > > > > > pick
> > > > > > > > > > from the list randomly.  Again, doing a curl to verify
> the
> > > > mirror
> > > > > > is
> > > > > > > up
> > > > > > > > > and
> > > > > > > > > > then doing a redirect.
> > > > > > > > > > - If we want to get fancy, we could do a reverse IP
> lookup
> > > and
> > > > > try
> > > > > > to
> > > > > > > > > match
> > > > > > > > > > the requester with their closest geographical mirror.
> > > > > > > > > >
> > > > > > > > > > Thoughts?
> > > > > > > > > >
> > > > > > > > > > *Will STEVENS*
> > > > > > > > > > Lead Developer
> > > > > > > > > >
> > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > > > > > chiradeepv@gmail.com
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > What steps are needed to set up a mirror? What does
> Infra
> > > > need
> > > > > to
> > > > > > > do?
> > > > > > > > > Has
> > > > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > > > > raja.pullela@accelerite.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi will,
> > > > > > > > > > > >
> > > > > > > > > > > > I believe, we didn’t get to close ‘getting a mirror
> on
> > > > > Apache’
> > > > > > > > > because
> > > > > > > > > > we
> > > > > > > > > > > > needed someone on the Apache Infra side to close
> this.
> > > BTW,
> > > > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages this?)
> has
> > > > > all/most
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > content.  Once we can close on the Apache mirror for
> > > > hosting
> > > > > > the
> > > > > > > > > > > content, I
> > > > > > > > > > > > can help assist getting the content there.
> > > > > > > > > > > >
> > > > > > > > > > > > For now, we have replicated the download.cloud.com
> > > content
> > > > > to
> > > > > > ‘
> > > > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > > > Also, we are working on a set of steps/procedure to
> > help
> > > > with
> > > > > > > this
> > > > > > > > > > > > change.  I will update everyone in about a week’s
> time
> > on
> > > > the
> > > > > > > > > details.
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Raja Pullela
> > > > > > > > > > > > Engineering Team,
> > > > > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > > > > >
> > > > > > > > > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on
> > > behalf
> > > > of
> > > > > > > Will
> > > > > > > > > > > > Stevens" <williamstevens@gmail.com on behalf of
> > > > > > > > > wstevens@cloudops.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > unfortunately the template mirror conversation got
> > caught
> > > > up
> > > > > in
> > > > > > > > > details
> > > > > > > > > > > and
> > > > > > > > > > > > nobody took the lead on implementing a solution.
> > > > > > > > > > > >
> > > > > > > > > > > > citrix has been pinging me every couple months to say
> > > > 'dude,
> > > > > we
> > > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > > remove the dependency on download.citrix.com', but i
> > > have
> > > > > not
> > > > > > > had
> > > > > > > > > the
> > > > > > > > > > > > cycles to get in and solve the problem.  the shutdown
> > of
> > > > that
> > > > > > is
> > > > > > > > > > imminent
> > > > > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > Lead Developer
> > > > > > > > > > > >
> > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Nathan,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Ideally, if you put the template location in (or
> use
> > a
> > > > > > template
> > > > > > > > > > defined
> > > > > > > > > > > > > in)  test_data.py then the actual location can be
> > > > > overridden
> > > > > > by
> > > > > > > > > > anyone
> > > > > > > > > > > > > testing.
> > > > > > > > > > > > >
> > > > > > > > > > > > > For Trillian, we've copied all of the templates
> that
> > > > people
> > > > > > > have
> > > > > > > > > > define
> > > > > > > > > > > > to
> > > > > > > > > > > > > a local repo and then replace the URLs in
> > test_data.py
> > > to
> > > > > > > reduce
> > > > > > > > > > > > bandwidth
> > > > > > > > > > > > > use and download times.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Ie:
> > > > > > > > > > > > >
> > > > > > > > > > > > >             "bootableIso":
> > > > > > > > > > > > >                 {
> > > > > > > > > > > > >                     "displaytext": "Test Bootable
> > ISO",
> > > > > > > > > > > > >                     "name": "testISO",
> > > > > > > > > > > > >                     "bootable": True,
> > > > > > > > > > > > >                     "ispublic": False,
> > > > > > > > > > > > >                     "url": "{{
> marvin_images_location
> > > > > > > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > > > > > >                     "ostype": 'Other Linux
> (64-bit)',
> > > > > > > > > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > > > > > > > > >         },
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I thought that we had come up with a solution for
> > > > > > > > > download.cloud.com
> > > > > > > > > > ,
> > > > > > > > > > > by
> > > > > > > > > > > > > having a mirrorlist hosted in Community Apache
> > 'space'
> > > > with
> > > > > > > > anyone
> > > > > > > > > > able
> > > > > > > > > > > > to
> > > > > > > > > > > > > out themselves forward as a mirror.
> > > > > > > > > > > > > But I must admit I lost track of whether anyone
> made
> > > the
> > > > > > > > requisite
> > > > > > > > > > > > changes
> > > > > > > > > > > > > in code....
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Kind regards,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Paul Angus
> > > > > > > > > > > > >
> > > > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > > > www.shapeblue.com
> > > > > > > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > > > > > > > @shapeblue
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > > From: williamstevens@gmail.com [mailto:
> > > > > williamstevens@gmail.
> > > > > > > com]
> > > > > > > > > On
> > > > > > > > > > > > > Behalf Of Will Stevens
> > > > > > > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > > > > > Subject: Re: Modern template hosting
> > > > > > > > > > > > >
> > > > > > > > > > > > > this is a hard questions.  in general, we should be
> > > > setting
> > > > > > up
> > > > > > > a
> > > > > > > > > > mirror
> > > > > > > > > > > > on
> > > > > > > > > > > > > some cloudstack/apache domain and then mirror to
> > other
> > > > > > provided
> > > > > > > > > > > > templates.
> > > > > > > > > > > > >
> > > > > > > > > > > > > we MUST come up with a solution to deprecate '
> > > > > > > download.cloud.com
> > > > > > > > ',
> > > > > > > > > > > that
> > > > > > > > > > > > > is going to be going away any day now.
> > > > > > > > > > > > >
> > > > > > > > > > > > > i don't know the right way to solve this to be
> > honest,
> > > > but
> > > > > if
> > > > > > > you
> > > > > > > > > > have
> > > > > > > > > > > > > ideas, i am willing to help.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > >
> > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> > > > > > > > njohnson@ena.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > So not to re-open a can of worms, but I’m in a
> > > > situation
> > > > > > > where
> > > > > > > > I
> > > > > > > > > > need
> > > > > > > > > > > > > > to come up with a Marvin component test that
> > depends
> > > > on a
> > > > > > > > > template
> > > > > > > > > > > > > > based on a kernel that’s relatively new, i.e.,
> > newer
> > > > than
> > > > > > > > Centos
> > > > > > > > > > 5.3
> > > > > > > > > > > /
> > > > > > > > > > > > > Ubuntu 10.04 .
> > > > > > > > > > > > > > I see openvm.eu has a suitable template (Ubuntu
> > > 16.0.4
> > > > > for
> > > > > > > > KVM),
> > > > > > > > > > but
> > > > > > > > > > > > > > from looking at the thread "Migrating CloudStack
> > > > content
> > > > > > from
> > > > > > > > > > > > > > download.cloud.com” it looks like there is
> > > resistance
> > > > to
> > > > > > > using
> > > > > > > > > > this
> > > > > > > > > > > at
> > > > > > > > > > > > > > least for hosting system vm templates over
> concerns
> > > of
> > > > > > > > > neutrality.
> > > > > > > > > > > > > > Would this be suitable for a component test?  If
> > not,
> > > > > what
> > > > > > > is a
> > > > > > > > > > > > “blessed”
> > > > > > > > > > > > > > template location?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks in advance!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Nathan Johnson
> > > > > > > > > > > > > > R&D Engineer
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > > > > > > Nashville, TN 37211
> > > > > > > > > > > > > > General Office: 615-312-6000
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > website | blog | support
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > DISCLAIMER
> > > > > > > > > > > > ==========
> > > > > > > > > > > > This e-mail may contain privileged and confidential
> > > > > information
> > > > > > > > which
> > > > > > > > > > is
> > > > > > > > > > > > the property of Accelerite, a Persistent Systems
> > > business.
> > > > It
> > > > > > is
> > > > > > > > > > intended
> > > > > > > > > > > > only for the use of the individual or entity to which
> > it
> > > is
> > > > > > > > > addressed.
> > > > > > > > > > If
> > > > > > > > > > > > you are not the intended recipient, you are not
> > > authorized
> > > > to
> > > > > > > read,
> > > > > > > > > > > retain,
> > > > > > > > > > > > copy, print, distribute or use this message. If you
> > have
> > > > > > received
> > > > > > > > > this
> > > > > > > > > > > > communication in error, please notify the sender and
> > > delete
> > > > > all
> > > > > > > > > copies
> > > > > > > > > > of
> > > > > > > > > > > > this message. Accelerite, a Persistent Systems
> business
> > > > does
> > > > > > not
> > > > > > > > > accept
> > > > > > > > > > > any
> > > > > > > > > > > > liability for virus infected mails.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rafael Weingärtner
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> > >
> >
>
>
>
> --
> Rafael Weingärtner
>

Re: Modern template hosting

Posted by Rafael Weingärtner <ra...@gmail.com>.

Agree with you.
We need to support the current working flow. And then, define the first
version that will start using the new approach.

On Mon, Feb 27, 2017 at 5:36 PM, Will Stevens <ws...@cloudops.com> wrote:

> I think we almost need a two pronged approach.
>
> 1) Get a solution in place which will enable us to document and serve
> templates for legacy systems.  I will work on this.
> 2) Discuss and understand how we SHOULD be handling this problem in the
> future and in what release we can expect it.
>
> I think we need to do both.  I think we should start to try to really
> understand what we want to deliver in (2) going forward.
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Mon, Feb 27, 2017 at 4:53 PM, Rafael Weingärtner <
> rafaelweingartner@gmail.com> wrote:
>
> > My worry is exactly with system VMs templates.
> >
> > Currently, we indicate administrators to download them from
> > http://cloudstack.apt-get.eu/systemvm/4.6/ [1]. However, the
> installation
> > docs do not mention the expected hashes for the file that is going to be
> > downloaded.
> > Also, I do not know the code that downloads system VMs templates (when
> > upgrading), but if the hash being checked is taken from the mirror used
> to
> > download the file; the only thing it checks is that if the download
> > finished successfully (no transmission errors). If we want to check
> > integrity, check that the template we created is untampered; we need to
> > host and serve the hash in a secure manner.
> >
> > [1]
> > http://docs.cloudstack.apache.org/projects/cloudstack-
> installation/en/4.9/
> > management-server/index.html#prepare-the-system-vm-template
> >
> >
> > On Mon, Feb 27, 2017 at 4:36 PM, Chiradeep Vittal <ch...@gmail.com>
> > wrote:
> >
> > > Hashes are checked (md5 IIRC) today.
> > > But given the issues, I think the project should steer away from
> hosting
> > > templates except the systemvm template.
> > >
> > > On Mon, Feb 27, 2017 at 1:31 PM, Rafael Weingärtner <
> > > rafaelweingartner@gmail.com> wrote:
> > >
> > > > Will, I think we could support different path structures. This can
> > > > facilitate different deployment of mirrors based on the structure the
> > > host
> > > > has.
> > > >
> > > > Could I add something else to the discussion? Have we discussed the
> > > > security impacts of setting up this mirrors approach?
> > > > I mean, if any of the mirrors gets corrupted (let`s say by a hacker),
> > and
> > > > the templates are injected with malicious code, an attacker could
> > > > potentially get un-monitored and unlimited access to a cloud
> > environment.
> > > >
> > > > If we assume that the mirror may get malicious (it is not that I do
> not
> > > > trust you guys, but bad things happen), we cannot host hashes there.
> > > Where
> > > > do you think we could store Sha512 or another hash type for these
> > > > templates? Could we host in the newly proposed Github repo or maybe
> > some
> > > > place in the ACS website?
> > > >
> > > > This would have an impact on clients (needing clear documentation)
> and
> > > our
> > > > code that automatically downloads System VM templates (does it check
> > > hashes
> > > > when automatically installing templates today? It may require
> > > > implementation changes).
> > > >
> > > > On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <wstevens@cloudops.com
> >
> > > > wrote:
> > > >
> > > > > so this is what I am looking to do.  Please let me know if you have
> > > > > suggestions for me or think I should be solving the problem a
> > different
> > > > > way.
> > > > >
> > > > > - We request a new Github repository from the ASF at:
> > > > > 'apache/cloudstack-mirror-list'
> > > > > - In this repository we track a text file in the 'gh-pages' branch
> > > with a
> > > > > list of valid download mirrors.
> > > > > - I build a binary to be hosted by the ASF (or at least with the
> ASF
> > > > > pointing a domain at the binary and I could potentially host it).
> We
> > > > will
> > > > > see how they want to handle the hosting of the binary.
> > > > >
> > > > > The binary would expose a web server which would behave as follows:
> > > > > - When the 'client' requests a download url the following flow is
> > > kicked
> > > > > off:
> > > > > -- The mirror list is queried from github (or from a static site
> > hosted
> > > > on
> > > > > asf, as we see fit).
> > > > > -- The Lat/Lon of the 'client' is determined based on their IP.
> > > > > -- The Lat/Lon for each of the 'mirror's is determined based on an
> IP
> > > > > lookup of the hostname.
> > > > > -- The closest geographical mirror is determined, the target is
> > > validated
> > > > > to be available and the user is redirected.
> > > > >
> > > > > Some questions I have right now:
> > > > > - Will every mirror have the same path structure to access the
> > > equivalent
> > > > > resources?
> > > > > - Should we support adding a path to the mirror url to specify the
> > path
> > > > to
> > > > > the base common path?
> > > > > -- Example: lets say the binary is hosted on 'dl.acs.com' and
> there
> > > are
> > > > > three mirrors 'abc.com', 'pqr.com/files' and 'xyx.com/downloads'.
> > > > > -- If the path being requested is '
> > > > > dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would result in
> > the
> > > > > following potential paths for the mirrors:
> > > > > -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > -- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'
> > > > >
> > > > > Does this all make sense?
> > > > >
> > > > > *Will STEVENS*
> > > > > Lead Developer
> > > > >
> > > > > <https://goo.gl/NYZ8KK>
> > > > >
> > > > > On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <
> > > chiradeepv@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > My bad. A few lines down, this has been added recently:
> > > > > >
> > > > > > this.request.setFollowRedirects(true);
> > > > > >
> > > > > > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <
> > > > williamstevens@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > OK. Thanks for the heads up.
> > > > > > >
> > > > > > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > chiradeepv@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Sounds workable. The downloader code in the SSVM won't follow
> > > > > > redirects I
> > > > > > > > think.
> > > > > > > > https://github.com/apache/cloudstack/blob/
> > > > > > 5511065fc20787619d9cd0444a65a3
> > > > > > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > > > > > HttpTemplateDownloader.java#L93
> > > > > > > > https://goo.gl/dSi0r5
> > > > > > > >  Might need to add
> > > > > > > > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > > > > > >
> > > > > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <
> > > > wstevens@cloudops.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > We haven't opened a ticket yet because we don't have a
> > strategy
> > > > > yet.
> > > > > > > > >
> > > > > > > > > What do you guys think of this:
> > > > > > > > > - We setup a new github repo in the 'apache' org which
> > consists
> > > > of
> > > > > a
> > > > > > > > single
> > > > > > > > > file with a list of active/supported mirrors.
> > > > > > > > > - I write a small web server, distributed as a binary,
> which
> > > can
> > > > be
> > > > > > > > hosted
> > > > > > > > > by ASF Infra.  This web server will query the current list
> of
> > > > > mirrors
> > > > > > > and
> > > > > > > > > will select one and then do a 302 redirect to that mirror.
> > > > > > > > >
> > > > > > > > > The act of 'choosing' a mirror could be done in a number of
> > > ways.
> > > > > > > > > - If we want to define an order, then it could just try
> from
> > > the
> > > > > top
> > > > > > of
> > > > > > > > the
> > > > > > > > > list and work its way down.  It would curl the target to
> make
> > > > sure
> > > > > it
> > > > > > > > gets
> > > > > > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > > > > > - Or, if we want to distribute the load across the mirrors,
> > we
> > > > > could
> > > > > > > pick
> > > > > > > > > from the list randomly.  Again, doing a curl to verify the
> > > mirror
> > > > > is
> > > > > > up
> > > > > > > > and
> > > > > > > > > then doing a redirect.
> > > > > > > > > - If we want to get fancy, we could do a reverse IP lookup
> > and
> > > > try
> > > > > to
> > > > > > > > match
> > > > > > > > > the requester with their closest geographical mirror.
> > > > > > > > >
> > > > > > > > > Thoughts?
> > > > > > > > >
> > > > > > > > > *Will STEVENS*
> > > > > > > > > Lead Developer
> > > > > > > > >
> > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > >
> > > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > > > > chiradeepv@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > What steps are needed to set up a mirror? What does Infra
> > > need
> > > > to
> > > > > > do?
> > > > > > > > Has
> > > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > > >
> > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > > > raja.pullela@accelerite.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi will,
> > > > > > > > > > >
> > > > > > > > > > > I believe, we didn’t get to close ‘getting a mirror on
> > > > Apache’
> > > > > > > > because
> > > > > > > > > we
> > > > > > > > > > > needed someone on the Apache Infra side to close this.
> > BTW,
> > > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
> > > > all/most
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > content.  Once we can close on the Apache mirror for
> > > hosting
> > > > > the
> > > > > > > > > > content, I
> > > > > > > > > > > can help assist getting the content there.
> > > > > > > > > > >
> > > > > > > > > > > For now, we have replicated the download.cloud.com
> > content
> > > > to
> > > > > ‘
> > > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > > Also, we are working on a set of steps/procedure to
> help
> > > with
> > > > > > this
> > > > > > > > > > > change.  I will update everyone in about a week’s time
> on
> > > the
> > > > > > > > details.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Raja Pullela
> > > > > > > > > > > Engineering Team,
> > > > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > > > >
> > > > > > > > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on
> > behalf
> > > of
> > > > > > Will
> > > > > > > > > > > Stevens" <williamstevens@gmail.com on behalf of
> > > > > > > > wstevens@cloudops.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > unfortunately the template mirror conversation got
> caught
> > > up
> > > > in
> > > > > > > > details
> > > > > > > > > > and
> > > > > > > > > > > nobody took the lead on implementing a solution.
> > > > > > > > > > >
> > > > > > > > > > > citrix has been pinging me every couple months to say
> > > 'dude,
> > > > we
> > > > > > > need
> > > > > > > > to
> > > > > > > > > > > remove the dependency on download.citrix.com', but i
> > have
> > > > not
> > > > > > had
> > > > > > > > the
> > > > > > > > > > > cycles to get in and solve the problem.  the shutdown
> of
> > > that
> > > > > is
> > > > > > > > > imminent
> > > > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > Lead Developer
> > > > > > > > > > >
> > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Nathan,
> > > > > > > > > > > >
> > > > > > > > > > > > Ideally, if you put the template location in (or use
> a
> > > > > template
> > > > > > > > > defined
> > > > > > > > > > > > in)  test_data.py then the actual location can be
> > > > overridden
> > > > > by
> > > > > > > > > anyone
> > > > > > > > > > > > testing.
> > > > > > > > > > > >
> > > > > > > > > > > > For Trillian, we've copied all of the templates that
> > > people
> > > > > > have
> > > > > > > > > define
> > > > > > > > > > > to
> > > > > > > > > > > > a local repo and then replace the URLs in
> test_data.py
> > to
> > > > > > reduce
> > > > > > > > > > > bandwidth
> > > > > > > > > > > > use and download times.
> > > > > > > > > > > >
> > > > > > > > > > > > Ie:
> > > > > > > > > > > >
> > > > > > > > > > > >             "bootableIso":
> > > > > > > > > > > >                 {
> > > > > > > > > > > >                     "displaytext": "Test Bootable
> ISO",
> > > > > > > > > > > >                     "name": "testISO",
> > > > > > > > > > > >                     "bootable": True,
> > > > > > > > > > > >                     "ispublic": False,
> > > > > > > > > > > >                     "url": "{{ marvin_images_location
> > > > > > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > > > > >                     "ostype": 'Other Linux (64-bit)',
> > > > > > > > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > > > > > > > >         },
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I thought that we had come up with a solution for
> > > > > > > > download.cloud.com
> > > > > > > > > ,
> > > > > > > > > > by
> > > > > > > > > > > > having a mirrorlist hosted in Community Apache
> 'space'
> > > with
> > > > > > > anyone
> > > > > > > > > able
> > > > > > > > > > > to
> > > > > > > > > > > > out themselves forward as a mirror.
> > > > > > > > > > > > But I must admit I lost track of whether anyone made
> > the
> > > > > > > requisite
> > > > > > > > > > > changes
> > > > > > > > > > > > in code....
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Kind regards,
> > > > > > > > > > > >
> > > > > > > > > > > > Paul Angus
> > > > > > > > > > > >
> > > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > > www.shapeblue.com
> > > > > > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > > > > > > @shapeblue
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > From: williamstevens@gmail.com [mailto:
> > > > williamstevens@gmail.
> > > > > > com]
> > > > > > > > On
> > > > > > > > > > > > Behalf Of Will Stevens
> > > > > > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > > > > Subject: Re: Modern template hosting
> > > > > > > > > > > >
> > > > > > > > > > > > this is a hard questions.  in general, we should be
> > > setting
> > > > > up
> > > > > > a
> > > > > > > > > mirror
> > > > > > > > > > > on
> > > > > > > > > > > > some cloudstack/apache domain and then mirror to
> other
> > > > > provided
> > > > > > > > > > > templates.
> > > > > > > > > > > >
> > > > > > > > > > > > we MUST come up with a solution to deprecate '
> > > > > > download.cloud.com
> > > > > > > ',
> > > > > > > > > > that
> > > > > > > > > > > > is going to be going away any day now.
> > > > > > > > > > > >
> > > > > > > > > > > > i don't know the right way to solve this to be
> honest,
> > > but
> > > > if
> > > > > > you
> > > > > > > > > have
> > > > > > > > > > > > ideas, i am willing to help.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > Lead Developer
> > > > > > > > > > > >
> > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> > > > > > > njohnson@ena.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > So not to re-open a can of worms, but I’m in a
> > > situation
> > > > > > where
> > > > > > > I
> > > > > > > > > need
> > > > > > > > > > > > > to come up with a Marvin component test that
> depends
> > > on a
> > > > > > > > template
> > > > > > > > > > > > > based on a kernel that’s relatively new, i.e.,
> newer
> > > than
> > > > > > > Centos
> > > > > > > > > 5.3
> > > > > > > > > > /
> > > > > > > > > > > > Ubuntu 10.04 .
> > > > > > > > > > > > > I see openvm.eu has a suitable template (Ubuntu
> > 16.0.4
> > > > for
> > > > > > > KVM),
> > > > > > > > > but
> > > > > > > > > > > > > from looking at the thread "Migrating CloudStack
> > > content
> > > > > from
> > > > > > > > > > > > > download.cloud.com” it looks like there is
> > resistance
> > > to
> > > > > > using
> > > > > > > > > this
> > > > > > > > > > at
> > > > > > > > > > > > > least for hosting system vm templates over concerns
> > of
> > > > > > > > neutrality.
> > > > > > > > > > > > > Would this be suitable for a component test?  If
> not,
> > > > what
> > > > > > is a
> > > > > > > > > > > “blessed”
> > > > > > > > > > > > > template location?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks in advance!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Nathan Johnson
> > > > > > > > > > > > > R&D Engineer
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > > > > > Nashville, TN 37211
> > > > > > > > > > > > > General Office: 615-312-6000
> > > > > > > > > > > > >
> > > > > > > > > > > > > website | blog | support
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > DISCLAIMER
> > > > > > > > > > > ==========
> > > > > > > > > > > This e-mail may contain privileged and confidential
> > > > information
> > > > > > > which
> > > > > > > > > is
> > > > > > > > > > > the property of Accelerite, a Persistent Systems
> > business.
> > > It
> > > > > is
> > > > > > > > > intended
> > > > > > > > > > > only for the use of the individual or entity to which
> it
> > is
> > > > > > > > addressed.
> > > > > > > > > If
> > > > > > > > > > > you are not the intended recipient, you are not
> > authorized
> > > to
> > > > > > read,
> > > > > > > > > > retain,
> > > > > > > > > > > copy, print, distribute or use this message. If you
> have
> > > > > received
> > > > > > > > this
> > > > > > > > > > > communication in error, please notify the sender and
> > delete
> > > > all
> > > > > > > > copies
> > > > > > > > > of
> > > > > > > > > > > this message. Accelerite, a Persistent Systems business
> > > does
> > > > > not
> > > > > > > > accept
> > > > > > > > > > any
> > > > > > > > > > > liability for virus infected mails.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Rafael Weingärtner
> > > >
> > >
> >
> >
> >
> > --
> > Rafael Weingärtner
> >
>



-- 
Rafael Weingärtner

Re: Modern template hosting

Posted by Will Stevens <ws...@cloudops.com>.

I think we almost need a two pronged approach.

1) Get a solution in place which will enable us to document and serve
templates for legacy systems.  I will work on this.
2) Discuss and understand how we SHOULD be handling this problem in the
future and in what release we can expect it.

I think we need to do both.  I think we should start to try to really
understand what we want to deliver in (2) going forward.

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Mon, Feb 27, 2017 at 4:53 PM, Rafael Weingärtner <
rafaelweingartner@gmail.com> wrote:

> My worry is exactly with system VMs templates.
>
> Currently, we indicate administrators to download them from
> http://cloudstack.apt-get.eu/systemvm/4.6/ [1]. However, the installation
> docs do not mention the expected hashes for the file that is going to be
> downloaded.
> Also, I do not know the code that downloads system VMs templates (when
> upgrading), but if the hash being checked is taken from the mirror used to
> download the file; the only thing it checks is that if the download
> finished successfully (no transmission errors). If we want to check
> integrity, check that the template we created is untampered; we need to
> host and serve the hash in a secure manner.
>
> [1]
> http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.9/
> management-server/index.html#prepare-the-system-vm-template
>
>
> On Mon, Feb 27, 2017 at 4:36 PM, Chiradeep Vittal <ch...@gmail.com>
> wrote:
>
> > Hashes are checked (md5 IIRC) today.
> > But given the issues, I think the project should steer away from hosting
> > templates except the systemvm template.
> >
> > On Mon, Feb 27, 2017 at 1:31 PM, Rafael Weingärtner <
> > rafaelweingartner@gmail.com> wrote:
> >
> > > Will, I think we could support different path structures. This can
> > > facilitate different deployment of mirrors based on the structure the
> > host
> > > has.
> > >
> > > Could I add something else to the discussion? Have we discussed the
> > > security impacts of setting up this mirrors approach?
> > > I mean, if any of the mirrors gets corrupted (let`s say by a hacker),
> and
> > > the templates are injected with malicious code, an attacker could
> > > potentially get un-monitored and unlimited access to a cloud
> environment.
> > >
> > > If we assume that the mirror may get malicious (it is not that I do not
> > > trust you guys, but bad things happen), we cannot host hashes there.
> > Where
> > > do you think we could store Sha512 or another hash type for these
> > > templates? Could we host in the newly proposed Github repo or maybe
> some
> > > place in the ACS website?
> > >
> > > This would have an impact on clients (needing clear documentation) and
> > our
> > > code that automatically downloads System VM templates (does it check
> > hashes
> > > when automatically installing templates today? It may require
> > > implementation changes).
> > >
> > > On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <ws...@cloudops.com>
> > > wrote:
> > >
> > > > so this is what I am looking to do.  Please let me know if you have
> > > > suggestions for me or think I should be solving the problem a
> different
> > > > way.
> > > >
> > > > - We request a new Github repository from the ASF at:
> > > > 'apache/cloudstack-mirror-list'
> > > > - In this repository we track a text file in the 'gh-pages' branch
> > with a
> > > > list of valid download mirrors.
> > > > - I build a binary to be hosted by the ASF (or at least with the ASF
> > > > pointing a domain at the binary and I could potentially host it).  We
> > > will
> > > > see how they want to handle the hosting of the binary.
> > > >
> > > > The binary would expose a web server which would behave as follows:
> > > > - When the 'client' requests a download url the following flow is
> > kicked
> > > > off:
> > > > -- The mirror list is queried from github (or from a static site
> hosted
> > > on
> > > > asf, as we see fit).
> > > > -- The Lat/Lon of the 'client' is determined based on their IP.
> > > > -- The Lat/Lon for each of the 'mirror's is determined based on an IP
> > > > lookup of the hostname.
> > > > -- The closest geographical mirror is determined, the target is
> > validated
> > > > to be available and the user is redirected.
> > > >
> > > > Some questions I have right now:
> > > > - Will every mirror have the same path structure to access the
> > equivalent
> > > > resources?
> > > > - Should we support adding a path to the mirror url to specify the
> path
> > > to
> > > > the base common path?
> > > > -- Example: lets say the binary is hosted on 'dl.acs.com' and there
> > are
> > > > three mirrors 'abc.com', 'pqr.com/files' and 'xyx.com/downloads'.
> > > > -- If the path being requested is '
> > > > dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would result in
> the
> > > > following potential paths for the mirrors:
> > > > -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> > > > -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> > > > -- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'
> > > >
> > > > Does this all make sense?
> > > >
> > > > *Will STEVENS*
> > > > Lead Developer
> > > >
> > > > <https://goo.gl/NYZ8KK>
> > > >
> > > > On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <
> > chiradeepv@gmail.com>
> > > > wrote:
> > > >
> > > > > My bad. A few lines down, this has been added recently:
> > > > >
> > > > > this.request.setFollowRedirects(true);
> > > > >
> > > > > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <
> > > williamstevens@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > OK. Thanks for the heads up.
> > > > > >
> > > > > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> chiradeepv@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Sounds workable. The downloader code in the SSVM won't follow
> > > > > redirects I
> > > > > > > think.
> > > > > > > https://github.com/apache/cloudstack/blob/
> > > > > 5511065fc20787619d9cd0444a65a3
> > > > > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > > > > HttpTemplateDownloader.java#L93
> > > > > > > https://goo.gl/dSi0r5
> > > > > > >  Might need to add
> > > > > > > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > > > > >
> > > > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <
> > > wstevens@cloudops.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > We haven't opened a ticket yet because we don't have a
> strategy
> > > > yet.
> > > > > > > >
> > > > > > > > What do you guys think of this:
> > > > > > > > - We setup a new github repo in the 'apache' org which
> consists
> > > of
> > > > a
> > > > > > > single
> > > > > > > > file with a list of active/supported mirrors.
> > > > > > > > - I write a small web server, distributed as a binary, which
> > can
> > > be
> > > > > > > hosted
> > > > > > > > by ASF Infra.  This web server will query the current list of
> > > > mirrors
> > > > > > and
> > > > > > > > will select one and then do a 302 redirect to that mirror.
> > > > > > > >
> > > > > > > > The act of 'choosing' a mirror could be done in a number of
> > ways.
> > > > > > > > - If we want to define an order, then it could just try from
> > the
> > > > top
> > > > > of
> > > > > > > the
> > > > > > > > list and work its way down.  It would curl the target to make
> > > sure
> > > > it
> > > > > > > gets
> > > > > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > > > > - Or, if we want to distribute the load across the mirrors,
> we
> > > > could
> > > > > > pick
> > > > > > > > from the list randomly.  Again, doing a curl to verify the
> > mirror
> > > > is
> > > > > up
> > > > > > > and
> > > > > > > > then doing a redirect.
> > > > > > > > - If we want to get fancy, we could do a reverse IP lookup
> and
> > > try
> > > > to
> > > > > > > match
> > > > > > > > the requester with their closest geographical mirror.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > *Will STEVENS*
> > > > > > > > Lead Developer
> > > > > > > >
> > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > >
> > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > > > chiradeepv@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > What steps are needed to set up a mirror? What does Infra
> > need
> > > to
> > > > > do?
> > > > > > > Has
> > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > >
> > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > > raja.pullela@accelerite.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi will,
> > > > > > > > > >
> > > > > > > > > > I believe, we didn’t get to close ‘getting a mirror on
> > > Apache’
> > > > > > > because
> > > > > > > > we
> > > > > > > > > > needed someone on the Apache Infra side to close this.
> BTW,
> > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
> > > all/most
> > > > > of
> > > > > > > the
> > > > > > > > > > content.  Once we can close on the Apache mirror for
> > hosting
> > > > the
> > > > > > > > > content, I
> > > > > > > > > > can help assist getting the content there.
> > > > > > > > > >
> > > > > > > > > > For now, we have replicated the download.cloud.com
> content
> > > to
> > > > ‘
> > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > Also, we are working on a set of steps/procedure to help
> > with
> > > > > this
> > > > > > > > > > change.  I will update everyone in about a week’s time on
> > the
> > > > > > > details.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Raja Pullela
> > > > > > > > > > Engineering Team,
> > > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > > >
> > > > > > > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on
> behalf
> > of
> > > > > Will
> > > > > > > > > > Stevens" <williamstevens@gmail.com on behalf of
> > > > > > > wstevens@cloudops.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > unfortunately the template mirror conversation got caught
> > up
> > > in
> > > > > > > details
> > > > > > > > > and
> > > > > > > > > > nobody took the lead on implementing a solution.
> > > > > > > > > >
> > > > > > > > > > citrix has been pinging me every couple months to say
> > 'dude,
> > > we
> > > > > > need
> > > > > > > to
> > > > > > > > > > remove the dependency on download.citrix.com', but i
> have
> > > not
> > > > > had
> > > > > > > the
> > > > > > > > > > cycles to get in and solve the problem.  the shutdown of
> > that
> > > > is
> > > > > > > > imminent
> > > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > *Will STEVENS*
> > > > > > > > > > Lead Developer
> > > > > > > > > >
> > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > > > > paul.angus@shapeblue.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Nathan,
> > > > > > > > > > >
> > > > > > > > > > > Ideally, if you put the template location in (or use a
> > > > template
> > > > > > > > defined
> > > > > > > > > > > in)  test_data.py then the actual location can be
> > > overridden
> > > > by
> > > > > > > > anyone
> > > > > > > > > > > testing.
> > > > > > > > > > >
> > > > > > > > > > > For Trillian, we've copied all of the templates that
> > people
> > > > > have
> > > > > > > > define
> > > > > > > > > > to
> > > > > > > > > > > a local repo and then replace the URLs in test_data.py
> to
> > > > > reduce
> > > > > > > > > > bandwidth
> > > > > > > > > > > use and download times.
> > > > > > > > > > >
> > > > > > > > > > > Ie:
> > > > > > > > > > >
> > > > > > > > > > >             "bootableIso":
> > > > > > > > > > >                 {
> > > > > > > > > > >                     "displaytext": "Test Bootable ISO",
> > > > > > > > > > >                     "name": "testISO",
> > > > > > > > > > >                     "bootable": True,
> > > > > > > > > > >                     "ispublic": False,
> > > > > > > > > > >                     "url": "{{ marvin_images_location
> > > > > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > > > >                     "ostype": 'Other Linux (64-bit)',
> > > > > > > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > > > > > > >         },
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I thought that we had come up with a solution for
> > > > > > > download.cloud.com
> > > > > > > > ,
> > > > > > > > > by
> > > > > > > > > > > having a mirrorlist hosted in Community Apache 'space'
> > with
> > > > > > anyone
> > > > > > > > able
> > > > > > > > > > to
> > > > > > > > > > > out themselves forward as a mirror.
> > > > > > > > > > > But I must admit I lost track of whether anyone made
> the
> > > > > > requisite
> > > > > > > > > > changes
> > > > > > > > > > > in code....
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Kind regards,
> > > > > > > > > > >
> > > > > > > > > > > Paul Angus
> > > > > > > > > > >
> > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > www.shapeblue.com
> > > > > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > > > > > @shapeblue
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: williamstevens@gmail.com [mailto:
> > > williamstevens@gmail.
> > > > > com]
> > > > > > > On
> > > > > > > > > > > Behalf Of Will Stevens
> > > > > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > > > Subject: Re: Modern template hosting
> > > > > > > > > > >
> > > > > > > > > > > this is a hard questions.  in general, we should be
> > setting
> > > > up
> > > > > a
> > > > > > > > mirror
> > > > > > > > > > on
> > > > > > > > > > > some cloudstack/apache domain and then mirror to other
> > > > provided
> > > > > > > > > > templates.
> > > > > > > > > > >
> > > > > > > > > > > we MUST come up with a solution to deprecate '
> > > > > download.cloud.com
> > > > > > ',
> > > > > > > > > that
> > > > > > > > > > > is going to be going away any day now.
> > > > > > > > > > >
> > > > > > > > > > > i don't know the right way to solve this to be honest,
> > but
> > > if
> > > > > you
> > > > > > > > have
> > > > > > > > > > > ideas, i am willing to help.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > Lead Developer
> > > > > > > > > > >
> > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> > > > > > njohnson@ena.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > So not to re-open a can of worms, but I’m in a
> > situation
> > > > > where
> > > > > > I
> > > > > > > > need
> > > > > > > > > > > > to come up with a Marvin component test that depends
> > on a
> > > > > > > template
> > > > > > > > > > > > based on a kernel that’s relatively new, i.e., newer
> > than
> > > > > > Centos
> > > > > > > > 5.3
> > > > > > > > > /
> > > > > > > > > > > Ubuntu 10.04 .
> > > > > > > > > > > > I see openvm.eu has a suitable template (Ubuntu
> 16.0.4
> > > for
> > > > > > KVM),
> > > > > > > > but
> > > > > > > > > > > > from looking at the thread "Migrating CloudStack
> > content
> > > > from
> > > > > > > > > > > > download.cloud.com” it looks like there is
> resistance
> > to
> > > > > using
> > > > > > > > this
> > > > > > > > > at
> > > > > > > > > > > > least for hosting system vm templates over concerns
> of
> > > > > > > neutrality.
> > > > > > > > > > > > Would this be suitable for a component test?  If not,
> > > what
> > > > > is a
> > > > > > > > > > “blessed”
> > > > > > > > > > > > template location?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks in advance!
> > > > > > > > > > > >
> > > > > > > > > > > > Nathan Johnson
> > > > > > > > > > > > R&D Engineer
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > > > > Nashville, TN 37211
> > > > > > > > > > > > General Office: 615-312-6000
> > > > > > > > > > > >
> > > > > > > > > > > > website | blog | support
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > DISCLAIMER
> > > > > > > > > > ==========
> > > > > > > > > > This e-mail may contain privileged and confidential
> > > information
> > > > > > which
> > > > > > > > is
> > > > > > > > > > the property of Accelerite, a Persistent Systems
> business.
> > It
> > > > is
> > > > > > > > intended
> > > > > > > > > > only for the use of the individual or entity to which it
> is
> > > > > > > addressed.
> > > > > > > > If
> > > > > > > > > > you are not the intended recipient, you are not
> authorized
> > to
> > > > > read,
> > > > > > > > > retain,
> > > > > > > > > > copy, print, distribute or use this message. If you have
> > > > received
> > > > > > > this
> > > > > > > > > > communication in error, please notify the sender and
> delete
> > > all
> > > > > > > copies
> > > > > > > > of
> > > > > > > > > > this message. Accelerite, a Persistent Systems business
> > does
> > > > not
> > > > > > > accept
> > > > > > > > > any
> > > > > > > > > > liability for virus infected mails.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> > >
> >
>
>
>
> --
> Rafael Weingärtner
>

Re: Modern template hosting

Posted by Rafael Weingärtner <ra...@gmail.com>.

My worry is exactly with system VMs templates.

Currently, we indicate administrators to download them from
http://cloudstack.apt-get.eu/systemvm/4.6/ [1]. However, the installation
docs do not mention the expected hashes for the file that is going to be
downloaded.
Also, I do not know the code that downloads system VMs templates (when
upgrading), but if the hash being checked is taken from the mirror used to
download the file; the only thing it checks is that if the download
finished successfully (no transmission errors). If we want to check
integrity, check that the template we created is untampered; we need to
host and serve the hash in a secure manner.

[1]
http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.9/management-server/index.html#prepare-the-system-vm-template


On Mon, Feb 27, 2017 at 4:36 PM, Chiradeep Vittal <ch...@gmail.com>
wrote:

> Hashes are checked (md5 IIRC) today.
> But given the issues, I think the project should steer away from hosting
> templates except the systemvm template.
>
> On Mon, Feb 27, 2017 at 1:31 PM, Rafael Weingärtner <
> rafaelweingartner@gmail.com> wrote:
>
> > Will, I think we could support different path structures. This can
> > facilitate different deployment of mirrors based on the structure the
> host
> > has.
> >
> > Could I add something else to the discussion? Have we discussed the
> > security impacts of setting up this mirrors approach?
> > I mean, if any of the mirrors gets corrupted (let`s say by a hacker), and
> > the templates are injected with malicious code, an attacker could
> > potentially get un-monitored and unlimited access to a cloud environment.
> >
> > If we assume that the mirror may get malicious (it is not that I do not
> > trust you guys, but bad things happen), we cannot host hashes there.
> Where
> > do you think we could store Sha512 or another hash type for these
> > templates? Could we host in the newly proposed Github repo or maybe some
> > place in the ACS website?
> >
> > This would have an impact on clients (needing clear documentation) and
> our
> > code that automatically downloads System VM templates (does it check
> hashes
> > when automatically installing templates today? It may require
> > implementation changes).
> >
> > On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <ws...@cloudops.com>
> > wrote:
> >
> > > so this is what I am looking to do.  Please let me know if you have
> > > suggestions for me or think I should be solving the problem a different
> > > way.
> > >
> > > - We request a new Github repository from the ASF at:
> > > 'apache/cloudstack-mirror-list'
> > > - In this repository we track a text file in the 'gh-pages' branch
> with a
> > > list of valid download mirrors.
> > > - I build a binary to be hosted by the ASF (or at least with the ASF
> > > pointing a domain at the binary and I could potentially host it).  We
> > will
> > > see how they want to handle the hosting of the binary.
> > >
> > > The binary would expose a web server which would behave as follows:
> > > - When the 'client' requests a download url the following flow is
> kicked
> > > off:
> > > -- The mirror list is queried from github (or from a static site hosted
> > on
> > > asf, as we see fit).
> > > -- The Lat/Lon of the 'client' is determined based on their IP.
> > > -- The Lat/Lon for each of the 'mirror's is determined based on an IP
> > > lookup of the hostname.
> > > -- The closest geographical mirror is determined, the target is
> validated
> > > to be available and the user is redirected.
> > >
> > > Some questions I have right now:
> > > - Will every mirror have the same path structure to access the
> equivalent
> > > resources?
> > > - Should we support adding a path to the mirror url to specify the path
> > to
> > > the base common path?
> > > -- Example: lets say the binary is hosted on 'dl.acs.com' and there
> are
> > > three mirrors 'abc.com', 'pqr.com/files' and 'xyx.com/downloads'.
> > > -- If the path being requested is '
> > > dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would result in the
> > > following potential paths for the mirrors:
> > > -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> > > -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> > > -- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'
> > >
> > > Does this all make sense?
> > >
> > > *Will STEVENS*
> > > Lead Developer
> > >
> > > <https://goo.gl/NYZ8KK>
> > >
> > > On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <
> chiradeepv@gmail.com>
> > > wrote:
> > >
> > > > My bad. A few lines down, this has been added recently:
> > > >
> > > > this.request.setFollowRedirects(true);
> > > >
> > > > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <
> > williamstevens@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > OK. Thanks for the heads up.
> > > > >
> > > > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <ch...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Sounds workable. The downloader code in the SSVM won't follow
> > > > redirects I
> > > > > > think.
> > > > > > https://github.com/apache/cloudstack/blob/
> > > > 5511065fc20787619d9cd0444a65a3
> > > > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > > > HttpTemplateDownloader.java#L93
> > > > > > https://goo.gl/dSi0r5
> > > > > >  Might need to add
> > > > > > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > > > >
> > > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <
> > wstevens@cloudops.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > We haven't opened a ticket yet because we don't have a strategy
> > > yet.
> > > > > > >
> > > > > > > What do you guys think of this:
> > > > > > > - We setup a new github repo in the 'apache' org which consists
> > of
> > > a
> > > > > > single
> > > > > > > file with a list of active/supported mirrors.
> > > > > > > - I write a small web server, distributed as a binary, which
> can
> > be
> > > > > > hosted
> > > > > > > by ASF Infra.  This web server will query the current list of
> > > mirrors
> > > > > and
> > > > > > > will select one and then do a 302 redirect to that mirror.
> > > > > > >
> > > > > > > The act of 'choosing' a mirror could be done in a number of
> ways.
> > > > > > > - If we want to define an order, then it could just try from
> the
> > > top
> > > > of
> > > > > > the
> > > > > > > list and work its way down.  It would curl the target to make
> > sure
> > > it
> > > > > > gets
> > > > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > > > - Or, if we want to distribute the load across the mirrors, we
> > > could
> > > > > pick
> > > > > > > from the list randomly.  Again, doing a curl to verify the
> mirror
> > > is
> > > > up
> > > > > > and
> > > > > > > then doing a redirect.
> > > > > > > - If we want to get fancy, we could do a reverse IP lookup and
> > try
> > > to
> > > > > > match
> > > > > > > the requester with their closest geographical mirror.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > *Will STEVENS*
> > > > > > > Lead Developer
> > > > > > >
> > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > >
> > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > > chiradeepv@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > What steps are needed to set up a mirror? What does Infra
> need
> > to
> > > > do?
> > > > > > Has
> > > > > > > > anybody filed a ticket with Infra?
> > > > > > > >
> > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > raja.pullela@accelerite.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi will,
> > > > > > > > >
> > > > > > > > > I believe, we didn’t get to close ‘getting a mirror on
> > Apache’
> > > > > > because
> > > > > > > we
> > > > > > > > > needed someone on the Apache Infra side to close this. BTW,
> > > > > > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
> > all/most
> > > > of
> > > > > > the
> > > > > > > > > content.  Once we can close on the Apache mirror for
> hosting
> > > the
> > > > > > > > content, I
> > > > > > > > > can help assist getting the content there.
> > > > > > > > >
> > > > > > > > > For now, we have replicated the download.cloud.com content
> > to
> > > ‘
> > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > Also, we are working on a set of steps/procedure to help
> with
> > > > this
> > > > > > > > > change.  I will update everyone in about a week’s time on
> the
> > > > > > details.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Raja Pullela
> > > > > > > > > Engineering Team,
> > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > >
> > > > > > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf
> of
> > > > Will
> > > > > > > > > Stevens" <williamstevens@gmail.com on behalf of
> > > > > > wstevens@cloudops.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > unfortunately the template mirror conversation got caught
> up
> > in
> > > > > > details
> > > > > > > > and
> > > > > > > > > nobody took the lead on implementing a solution.
> > > > > > > > >
> > > > > > > > > citrix has been pinging me every couple months to say
> 'dude,
> > we
> > > > > need
> > > > > > to
> > > > > > > > > remove the dependency on download.citrix.com', but i have
> > not
> > > > had
> > > > > > the
> > > > > > > > > cycles to get in and solve the problem.  the shutdown of
> that
> > > is
> > > > > > > imminent
> > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *Will STEVENS*
> > > > > > > > > Lead Developer
> > > > > > > > >
> > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > >
> > > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > > > paul.angus@shapeblue.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Nathan,
> > > > > > > > > >
> > > > > > > > > > Ideally, if you put the template location in (or use a
> > > template
> > > > > > > defined
> > > > > > > > > > in)  test_data.py then the actual location can be
> > overridden
> > > by
> > > > > > > anyone
> > > > > > > > > > testing.
> > > > > > > > > >
> > > > > > > > > > For Trillian, we've copied all of the templates that
> people
> > > > have
> > > > > > > define
> > > > > > > > > to
> > > > > > > > > > a local repo and then replace the URLs in test_data.py to
> > > > reduce
> > > > > > > > > bandwidth
> > > > > > > > > > use and download times.
> > > > > > > > > >
> > > > > > > > > > Ie:
> > > > > > > > > >
> > > > > > > > > >             "bootableIso":
> > > > > > > > > >                 {
> > > > > > > > > >                     "displaytext": "Test Bootable ISO",
> > > > > > > > > >                     "name": "testISO",
> > > > > > > > > >                     "bootable": True,
> > > > > > > > > >                     "ispublic": False,
> > > > > > > > > >                     "url": "{{ marvin_images_location
> > > > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > > >                     "ostype": 'Other Linux (64-bit)',
> > > > > > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > > > > > >         },
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I thought that we had come up with a solution for
> > > > > > download.cloud.com
> > > > > > > ,
> > > > > > > > by
> > > > > > > > > > having a mirrorlist hosted in Community Apache 'space'
> with
> > > > > anyone
> > > > > > > able
> > > > > > > > > to
> > > > > > > > > > out themselves forward as a mirror.
> > > > > > > > > > But I must admit I lost track of whether anyone made the
> > > > > requisite
> > > > > > > > > changes
> > > > > > > > > > in code....
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Kind regards,
> > > > > > > > > >
> > > > > > > > > > Paul Angus
> > > > > > > > > >
> > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > www.shapeblue.com
> > > > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > > > > @shapeblue
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: williamstevens@gmail.com [mailto:
> > williamstevens@gmail.
> > > > com]
> > > > > > On
> > > > > > > > > > Behalf Of Will Stevens
> > > > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > > Subject: Re: Modern template hosting
> > > > > > > > > >
> > > > > > > > > > this is a hard questions.  in general, we should be
> setting
> > > up
> > > > a
> > > > > > > mirror
> > > > > > > > > on
> > > > > > > > > > some cloudstack/apache domain and then mirror to other
> > > provided
> > > > > > > > > templates.
> > > > > > > > > >
> > > > > > > > > > we MUST come up with a solution to deprecate '
> > > > download.cloud.com
> > > > > ',
> > > > > > > > that
> > > > > > > > > > is going to be going away any day now.
> > > > > > > > > >
> > > > > > > > > > i don't know the right way to solve this to be honest,
> but
> > if
> > > > you
> > > > > > > have
> > > > > > > > > > ideas, i am willing to help.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > *Will STEVENS*
> > > > > > > > > > Lead Developer
> > > > > > > > > >
> > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> > > > > njohnson@ena.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > So not to re-open a can of worms, but I’m in a
> situation
> > > > where
> > > > > I
> > > > > > > need
> > > > > > > > > > > to come up with a Marvin component test that depends
> on a
> > > > > > template
> > > > > > > > > > > based on a kernel that’s relatively new, i.e., newer
> than
> > > > > Centos
> > > > > > > 5.3
> > > > > > > > /
> > > > > > > > > > Ubuntu 10.04 .
> > > > > > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4
> > for
> > > > > KVM),
> > > > > > > but
> > > > > > > > > > > from looking at the thread "Migrating CloudStack
> content
> > > from
> > > > > > > > > > > download.cloud.com” it looks like there is resistance
> to
> > > > using
> > > > > > > this
> > > > > > > > at
> > > > > > > > > > > least for hosting system vm templates over concerns of
> > > > > > neutrality.
> > > > > > > > > > > Would this be suitable for a component test?  If not,
> > what
> > > > is a
> > > > > > > > > “blessed”
> > > > > > > > > > > template location?
> > > > > > > > > > >
> > > > > > > > > > > Thanks in advance!
> > > > > > > > > > >
> > > > > > > > > > > Nathan Johnson
> > > > > > > > > > > R&D Engineer
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > > > Nashville, TN 37211
> > > > > > > > > > > General Office: 615-312-6000
> > > > > > > > > > >
> > > > > > > > > > > website | blog | support
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > DISCLAIMER
> > > > > > > > > ==========
> > > > > > > > > This e-mail may contain privileged and confidential
> > information
> > > > > which
> > > > > > > is
> > > > > > > > > the property of Accelerite, a Persistent Systems business.
> It
> > > is
> > > > > > > intended
> > > > > > > > > only for the use of the individual or entity to which it is
> > > > > > addressed.
> > > > > > > If
> > > > > > > > > you are not the intended recipient, you are not authorized
> to
> > > > read,
> > > > > > > > retain,
> > > > > > > > > copy, print, distribute or use this message. If you have
> > > received
> > > > > > this
> > > > > > > > > communication in error, please notify the sender and delete
> > all
> > > > > > copies
> > > > > > > of
> > > > > > > > > this message. Accelerite, a Persistent Systems business
> does
> > > not
> > > > > > accept
> > > > > > > > any
> > > > > > > > > liability for virus infected mails.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Rafael Weingärtner
> >
>



-- 
Rafael Weingärtner

Re: Modern template hosting

Posted by Chiradeep Vittal <ch...@gmail.com>.

Hashes are checked (md5 IIRC) today.
But given the issues, I think the project should steer away from hosting
templates except the systemvm template.

On Mon, Feb 27, 2017 at 1:31 PM, Rafael Weingärtner <
rafaelweingartner@gmail.com> wrote:

> Will, I think we could support different path structures. This can
> facilitate different deployment of mirrors based on the structure the host
> has.
>
> Could I add something else to the discussion? Have we discussed the
> security impacts of setting up this mirrors approach?
> I mean, if any of the mirrors gets corrupted (let`s say by a hacker), and
> the templates are injected with malicious code, an attacker could
> potentially get un-monitored and unlimited access to a cloud environment.
>
> If we assume that the mirror may get malicious (it is not that I do not
> trust you guys, but bad things happen), we cannot host hashes there. Where
> do you think we could store Sha512 or another hash type for these
> templates? Could we host in the newly proposed Github repo or maybe some
> place in the ACS website?
>
> This would have an impact on clients (needing clear documentation) and our
> code that automatically downloads System VM templates (does it check hashes
> when automatically installing templates today? It may require
> implementation changes).
>
> On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <ws...@cloudops.com>
> wrote:
>
> > so this is what I am looking to do.  Please let me know if you have
> > suggestions for me or think I should be solving the problem a different
> > way.
> >
> > - We request a new Github repository from the ASF at:
> > 'apache/cloudstack-mirror-list'
> > - In this repository we track a text file in the 'gh-pages' branch with a
> > list of valid download mirrors.
> > - I build a binary to be hosted by the ASF (or at least with the ASF
> > pointing a domain at the binary and I could potentially host it).  We
> will
> > see how they want to handle the hosting of the binary.
> >
> > The binary would expose a web server which would behave as follows:
> > - When the 'client' requests a download url the following flow is kicked
> > off:
> > -- The mirror list is queried from github (or from a static site hosted
> on
> > asf, as we see fit).
> > -- The Lat/Lon of the 'client' is determined based on their IP.
> > -- The Lat/Lon for each of the 'mirror's is determined based on an IP
> > lookup of the hostname.
> > -- The closest geographical mirror is determined, the target is validated
> > to be available and the user is redirected.
> >
> > Some questions I have right now:
> > - Will every mirror have the same path structure to access the equivalent
> > resources?
> > - Should we support adding a path to the mirror url to specify the path
> to
> > the base common path?
> > -- Example: lets say the binary is hosted on 'dl.acs.com' and there are
> > three mirrors 'abc.com', 'pqr.com/files' and 'xyx.com/downloads'.
> > -- If the path being requested is '
> > dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would result in the
> > following potential paths for the mirrors:
> > -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> > -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> > -- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'
> >
> > Does this all make sense?
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <ch...@gmail.com>
> > wrote:
> >
> > > My bad. A few lines down, this has been added recently:
> > >
> > > this.request.setFollowRedirects(true);
> > >
> > > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <
> williamstevens@gmail.com
> > >
> > > wrote:
> > >
> > > > OK. Thanks for the heads up.
> > > >
> > > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <ch...@gmail.com>
> > > wrote:
> > > >
> > > > > Sounds workable. The downloader code in the SSVM won't follow
> > > redirects I
> > > > > think.
> > > > > https://github.com/apache/cloudstack/blob/
> > > 5511065fc20787619d9cd0444a65a3
> > > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > > HttpTemplateDownloader.java#L93
> > > > > https://goo.gl/dSi0r5
> > > > >  Might need to add
> > > > > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > > >
> > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <
> wstevens@cloudops.com
> > >
> > > > > wrote:
> > > > >
> > > > > > We haven't opened a ticket yet because we don't have a strategy
> > yet.
> > > > > >
> > > > > > What do you guys think of this:
> > > > > > - We setup a new github repo in the 'apache' org which consists
> of
> > a
> > > > > single
> > > > > > file with a list of active/supported mirrors.
> > > > > > - I write a small web server, distributed as a binary, which can
> be
> > > > > hosted
> > > > > > by ASF Infra.  This web server will query the current list of
> > mirrors
> > > > and
> > > > > > will select one and then do a 302 redirect to that mirror.
> > > > > >
> > > > > > The act of 'choosing' a mirror could be done in a number of ways.
> > > > > > - If we want to define an order, then it could just try from the
> > top
> > > of
> > > > > the
> > > > > > list and work its way down.  It would curl the target to make
> sure
> > it
> > > > > gets
> > > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > > - Or, if we want to distribute the load across the mirrors, we
> > could
> > > > pick
> > > > > > from the list randomly.  Again, doing a curl to verify the mirror
> > is
> > > up
> > > > > and
> > > > > > then doing a redirect.
> > > > > > - If we want to get fancy, we could do a reverse IP lookup and
> try
> > to
> > > > > match
> > > > > > the requester with their closest geographical mirror.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > *Will STEVENS*
> > > > > > Lead Developer
> > > > > >
> > > > > > <https://goo.gl/NYZ8KK>
> > > > > >
> > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > chiradeepv@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > What steps are needed to set up a mirror? What does Infra need
> to
> > > do?
> > > > > Has
> > > > > > > anybody filed a ticket with Infra?
> > > > > > >
> > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > raja.pullela@accelerite.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi will,
> > > > > > > >
> > > > > > > > I believe, we didn’t get to close ‘getting a mirror on
> Apache’
> > > > > because
> > > > > > we
> > > > > > > > needed someone on the Apache Infra side to close this. BTW,
> > > > > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
> all/most
> > > of
> > > > > the
> > > > > > > > content.  Once we can close on the Apache mirror for hosting
> > the
> > > > > > > content, I
> > > > > > > > can help assist getting the content there.
> > > > > > > >
> > > > > > > > For now, we have replicated the download.cloud.com content
> to
> > ‘
> > > > > > > > s3.download.accelerite.com’.
> > > > > > > > Also, we are working on a set of steps/procedure to help with
> > > this
> > > > > > > > change.  I will update everyone in about a week’s time on the
> > > > > details.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Raja Pullela
> > > > > > > > Engineering Team,
> > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > Santa Clara, CA, 95054
> > > > > > > >
> > > > > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of
> > > Will
> > > > > > > > Stevens" <williamstevens@gmail.com on behalf of
> > > > > wstevens@cloudops.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > unfortunately the template mirror conversation got caught up
> in
> > > > > details
> > > > > > > and
> > > > > > > > nobody took the lead on implementing a solution.
> > > > > > > >
> > > > > > > > citrix has been pinging me every couple months to say 'dude,
> we
> > > > need
> > > > > to
> > > > > > > > remove the dependency on download.citrix.com', but i have
> not
> > > had
> > > > > the
> > > > > > > > cycles to get in and solve the problem.  the shutdown of that
> > is
> > > > > > imminent
> > > > > > > > right now, so we need to solve it asap.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *Will STEVENS*
> > > > > > > > Lead Developer
> > > > > > > >
> > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > >
> > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > > paul.angus@shapeblue.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Nathan,
> > > > > > > > >
> > > > > > > > > Ideally, if you put the template location in (or use a
> > template
> > > > > > defined
> > > > > > > > > in)  test_data.py then the actual location can be
> overridden
> > by
> > > > > > anyone
> > > > > > > > > testing.
> > > > > > > > >
> > > > > > > > > For Trillian, we've copied all of the templates that people
> > > have
> > > > > > define
> > > > > > > > to
> > > > > > > > > a local repo and then replace the URLs in test_data.py to
> > > reduce
> > > > > > > > bandwidth
> > > > > > > > > use and download times.
> > > > > > > > >
> > > > > > > > > Ie:
> > > > > > > > >
> > > > > > > > >             "bootableIso":
> > > > > > > > >                 {
> > > > > > > > >                     "displaytext": "Test Bootable ISO",
> > > > > > > > >                     "name": "testISO",
> > > > > > > > >                     "bootable": True,
> > > > > > > > >                     "ispublic": False,
> > > > > > > > >                     "url": "{{ marvin_images_location
> > > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > >                     "ostype": 'Other Linux (64-bit)',
> > > > > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > > > > >         },
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I thought that we had come up with a solution for
> > > > > download.cloud.com
> > > > > > ,
> > > > > > > by
> > > > > > > > > having a mirrorlist hosted in Community Apache 'space' with
> > > > anyone
> > > > > > able
> > > > > > > > to
> > > > > > > > > out themselves forward as a mirror.
> > > > > > > > > But I must admit I lost track of whether anyone made the
> > > > requisite
> > > > > > > > changes
> > > > > > > > > in code....
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Kind regards,
> > > > > > > > >
> > > > > > > > > Paul Angus
> > > > > > > > >
> > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > www.shapeblue.com
> > > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > > > @shapeblue
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: williamstevens@gmail.com [mailto:
> williamstevens@gmail.
> > > com]
> > > > > On
> > > > > > > > > Behalf Of Will Stevens
> > > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > Subject: Re: Modern template hosting
> > > > > > > > >
> > > > > > > > > this is a hard questions.  in general, we should be setting
> > up
> > > a
> > > > > > mirror
> > > > > > > > on
> > > > > > > > > some cloudstack/apache domain and then mirror to other
> > provided
> > > > > > > > templates.
> > > > > > > > >
> > > > > > > > > we MUST come up with a solution to deprecate '
> > > download.cloud.com
> > > > ',
> > > > > > > that
> > > > > > > > > is going to be going away any day now.
> > > > > > > > >
> > > > > > > > > i don't know the right way to solve this to be honest, but
> if
> > > you
> > > > > > have
> > > > > > > > > ideas, i am willing to help.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *Will STEVENS*
> > > > > > > > > Lead Developer
> > > > > > > > >
> > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > >
> > > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> > > > njohnson@ena.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > So not to re-open a can of worms, but I’m in a situation
> > > where
> > > > I
> > > > > > need
> > > > > > > > > > to come up with a Marvin component test that depends on a
> > > > > template
> > > > > > > > > > based on a kernel that’s relatively new, i.e., newer than
> > > > Centos
> > > > > > 5.3
> > > > > > > /
> > > > > > > > > Ubuntu 10.04 .
> > > > > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4
> for
> > > > KVM),
> > > > > > but
> > > > > > > > > > from looking at the thread "Migrating CloudStack content
> > from
> > > > > > > > > > download.cloud.com” it looks like there is resistance to
> > > using
> > > > > > this
> > > > > > > at
> > > > > > > > > > least for hosting system vm templates over concerns of
> > > > > neutrality.
> > > > > > > > > > Would this be suitable for a component test?  If not,
> what
> > > is a
> > > > > > > > “blessed”
> > > > > > > > > > template location?
> > > > > > > > > >
> > > > > > > > > > Thanks in advance!
> > > > > > > > > >
> > > > > > > > > > Nathan Johnson
> > > > > > > > > > R&D Engineer
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > > Nashville, TN 37211
> > > > > > > > > > General Office: 615-312-6000
> > > > > > > > > >
> > > > > > > > > > website | blog | support
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > DISCLAIMER
> > > > > > > > ==========
> > > > > > > > This e-mail may contain privileged and confidential
> information
> > > > which
> > > > > > is
> > > > > > > > the property of Accelerite, a Persistent Systems business. It
> > is
> > > > > > intended
> > > > > > > > only for the use of the individual or entity to which it is
> > > > > addressed.
> > > > > > If
> > > > > > > > you are not the intended recipient, you are not authorized to
> > > read,
> > > > > > > retain,
> > > > > > > > copy, print, distribute or use this message. If you have
> > received
> > > > > this
> > > > > > > > communication in error, please notify the sender and delete
> all
> > > > > copies
> > > > > > of
> > > > > > > > this message. Accelerite, a Persistent Systems business does
> > not
> > > > > accept
> > > > > > > any
> > > > > > > > liability for virus infected mails.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Rafael Weingärtner
>

Re: Modern template hosting

Posted by Rafael Weingärtner <ra...@gmail.com>.

Will, I think we could support different path structures. This can
facilitate different deployment of mirrors based on the structure the host
has.

Could I add something else to the discussion? Have we discussed the
security impacts of setting up this mirrors approach?
I mean, if any of the mirrors gets corrupted (let`s say by a hacker), and
the templates are injected with malicious code, an attacker could
potentially get un-monitored and unlimited access to a cloud environment.

If we assume that the mirror may get malicious (it is not that I do not
trust you guys, but bad things happen), we cannot host hashes there. Where
do you think we could store Sha512 or another hash type for these
templates? Could we host in the newly proposed Github repo or maybe some
place in the ACS website?

This would have an impact on clients (needing clear documentation) and our
code that automatically downloads System VM templates (does it check hashes
when automatically installing templates today? It may require
implementation changes).

On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <ws...@cloudops.com> wrote:

> so this is what I am looking to do.  Please let me know if you have
> suggestions for me or think I should be solving the problem a different
> way.
>
> - We request a new Github repository from the ASF at:
> 'apache/cloudstack-mirror-list'
> - In this repository we track a text file in the 'gh-pages' branch with a
> list of valid download mirrors.
> - I build a binary to be hosted by the ASF (or at least with the ASF
> pointing a domain at the binary and I could potentially host it).  We will
> see how they want to handle the hosting of the binary.
>
> The binary would expose a web server which would behave as follows:
> - When the 'client' requests a download url the following flow is kicked
> off:
> -- The mirror list is queried from github (or from a static site hosted on
> asf, as we see fit).
> -- The Lat/Lon of the 'client' is determined based on their IP.
> -- The Lat/Lon for each of the 'mirror's is determined based on an IP
> lookup of the hostname.
> -- The closest geographical mirror is determined, the target is validated
> to be available and the user is redirected.
>
> Some questions I have right now:
> - Will every mirror have the same path structure to access the equivalent
> resources?
> - Should we support adding a path to the mirror url to specify the path to
> the base common path?
> -- Example: lets say the binary is hosted on 'dl.acs.com' and there are
> three mirrors 'abc.com', 'pqr.com/files' and 'xyx.com/downloads'.
> -- If the path being requested is '
> dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would result in the
> following potential paths for the mirrors:
> -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> -- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'
>
> Does this all make sense?
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <ch...@gmail.com>
> wrote:
>
> > My bad. A few lines down, this has been added recently:
> >
> > this.request.setFollowRedirects(true);
> >
> > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <williamstevens@gmail.com
> >
> > wrote:
> >
> > > OK. Thanks for the heads up.
> > >
> > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <ch...@gmail.com>
> > wrote:
> > >
> > > > Sounds workable. The downloader code in the SSVM won't follow
> > redirects I
> > > > think.
> > > > https://github.com/apache/cloudstack/blob/
> > 5511065fc20787619d9cd0444a65a3
> > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > HttpTemplateDownloader.java#L93
> > > > https://goo.gl/dSi0r5
> > > >  Might need to add
> > > > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > >
> > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <wstevens@cloudops.com
> >
> > > > wrote:
> > > >
> > > > > We haven't opened a ticket yet because we don't have a strategy
> yet.
> > > > >
> > > > > What do you guys think of this:
> > > > > - We setup a new github repo in the 'apache' org which consists of
> a
> > > > single
> > > > > file with a list of active/supported mirrors.
> > > > > - I write a small web server, distributed as a binary, which can be
> > > > hosted
> > > > > by ASF Infra.  This web server will query the current list of
> mirrors
> > > and
> > > > > will select one and then do a 302 redirect to that mirror.
> > > > >
> > > > > The act of 'choosing' a mirror could be done in a number of ways.
> > > > > - If we want to define an order, then it could just try from the
> top
> > of
> > > > the
> > > > > list and work its way down.  It would curl the target to make sure
> it
> > > > gets
> > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > - Or, if we want to distribute the load across the mirrors, we
> could
> > > pick
> > > > > from the list randomly.  Again, doing a curl to verify the mirror
> is
> > up
> > > > and
> > > > > then doing a redirect.
> > > > > - If we want to get fancy, we could do a reverse IP lookup and try
> to
> > > > match
> > > > > the requester with their closest geographical mirror.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > *Will STEVENS*
> > > > > Lead Developer
> > > > >
> > > > > <https://goo.gl/NYZ8KK>
> > > > >
> > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > chiradeepv@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > What steps are needed to set up a mirror? What does Infra need to
> > do?
> > > > Has
> > > > > > anybody filed a ticket with Infra?
> > > > > >
> > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > raja.pullela@accelerite.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi will,
> > > > > > >
> > > > > > > I believe, we didn’t get to close ‘getting a mirror on Apache’
> > > > because
> > > > > we
> > > > > > > needed someone on the Apache Infra side to close this. BTW,
> > > > > > > cloudstack-apt.get.eu (I think Nux manages this?) has all/most
> > of
> > > > the
> > > > > > > content.  Once we can close on the Apache mirror for hosting
> the
> > > > > > content, I
> > > > > > > can help assist getting the content there.
> > > > > > >
> > > > > > > For now, we have replicated the download.cloud.com content to
> ‘
> > > > > > > s3.download.accelerite.com’.
> > > > > > > Also, we are working on a set of steps/procedure to help with
> > this
> > > > > > > change.  I will update everyone in about a week’s time on the
> > > > details.
> > > > > > >
> > > > > > > Best,
> > > > > > > Raja Pullela
> > > > > > > Engineering Team,
> > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > Santa Clara, CA, 95054
> > > > > > >
> > > > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of
> > Will
> > > > > > > Stevens" <williamstevens@gmail.com on behalf of
> > > > wstevens@cloudops.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > unfortunately the template mirror conversation got caught up in
> > > > details
> > > > > > and
> > > > > > > nobody took the lead on implementing a solution.
> > > > > > >
> > > > > > > citrix has been pinging me every couple months to say 'dude, we
> > > need
> > > > to
> > > > > > > remove the dependency on download.citrix.com', but i have not
> > had
> > > > the
> > > > > > > cycles to get in and solve the problem.  the shutdown of that
> is
> > > > > imminent
> > > > > > > right now, so we need to solve it asap.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *Will STEVENS*
> > > > > > > Lead Developer
> > > > > > >
> > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > >
> > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > paul.angus@shapeblue.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Nathan,
> > > > > > > >
> > > > > > > > Ideally, if you put the template location in (or use a
> template
> > > > > defined
> > > > > > > > in)  test_data.py then the actual location can be overridden
> by
> > > > > anyone
> > > > > > > > testing.
> > > > > > > >
> > > > > > > > For Trillian, we've copied all of the templates that people
> > have
> > > > > define
> > > > > > > to
> > > > > > > > a local repo and then replace the URLs in test_data.py to
> > reduce
> > > > > > > bandwidth
> > > > > > > > use and download times.
> > > > > > > >
> > > > > > > > Ie:
> > > > > > > >
> > > > > > > >             "bootableIso":
> > > > > > > >                 {
> > > > > > > >                     "displaytext": "Test Bootable ISO",
> > > > > > > >                     "name": "testISO",
> > > > > > > >                     "bootable": True,
> > > > > > > >                     "ispublic": False,
> > > > > > > >                     "url": "{{ marvin_images_location
> > > > > > > > }}/TinyCore-current.iso",
> > > > > > > >                     "ostype": 'Other Linux (64-bit)',
> > > > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > > > >         },
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > I thought that we had come up with a solution for
> > > > download.cloud.com
> > > > > ,
> > > > > > by
> > > > > > > > having a mirrorlist hosted in Community Apache 'space' with
> > > anyone
> > > > > able
> > > > > > > to
> > > > > > > > out themselves forward as a mirror.
> > > > > > > > But I must admit I lost track of whether anyone made the
> > > requisite
> > > > > > > changes
> > > > > > > > in code....
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Kind regards,
> > > > > > > >
> > > > > > > > Paul Angus
> > > > > > > >
> > > > > > > > paul.angus@shapeblue.com
> > > > > > > > www.shapeblue.com
> > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > > @shapeblue
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: williamstevens@gmail.com [mailto:williamstevens@gmail.
> > com]
> > > > On
> > > > > > > > Behalf Of Will Stevens
> > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > Subject: Re: Modern template hosting
> > > > > > > >
> > > > > > > > this is a hard questions.  in general, we should be setting
> up
> > a
> > > > > mirror
> > > > > > > on
> > > > > > > > some cloudstack/apache domain and then mirror to other
> provided
> > > > > > > templates.
> > > > > > > >
> > > > > > > > we MUST come up with a solution to deprecate '
> > download.cloud.com
> > > ',
> > > > > > that
> > > > > > > > is going to be going away any day now.
> > > > > > > >
> > > > > > > > i don't know the right way to solve this to be honest, but if
> > you
> > > > > have
> > > > > > > > ideas, i am willing to help.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *Will STEVENS*
> > > > > > > > Lead Developer
> > > > > > > >
> > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > >
> > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> > > njohnson@ena.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > So not to re-open a can of worms, but I’m in a situation
> > where
> > > I
> > > > > need
> > > > > > > > > to come up with a Marvin component test that depends on a
> > > > template
> > > > > > > > > based on a kernel that’s relatively new, i.e., newer than
> > > Centos
> > > > > 5.3
> > > > > > /
> > > > > > > > Ubuntu 10.04 .
> > > > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for
> > > KVM),
> > > > > but
> > > > > > > > > from looking at the thread "Migrating CloudStack content
> from
> > > > > > > > > download.cloud.com” it looks like there is resistance to
> > using
> > > > > this
> > > > > > at
> > > > > > > > > least for hosting system vm templates over concerns of
> > > > neutrality.
> > > > > > > > > Would this be suitable for a component test?  If not, what
> > is a
> > > > > > > “blessed”
> > > > > > > > > template location?
> > > > > > > > >
> > > > > > > > > Thanks in advance!
> > > > > > > > >
> > > > > > > > > Nathan Johnson
> > > > > > > > > R&D Engineer
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > Nashville, TN 37211
> > > > > > > > > General Office: 615-312-6000
> > > > > > > > >
> > > > > > > > > website | blog | support
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > DISCLAIMER
> > > > > > > ==========
> > > > > > > This e-mail may contain privileged and confidential information
> > > which
> > > > > is
> > > > > > > the property of Accelerite, a Persistent Systems business. It
> is
> > > > > intended
> > > > > > > only for the use of the individual or entity to which it is
> > > > addressed.
> > > > > If
> > > > > > > you are not the intended recipient, you are not authorized to
> > read,
> > > > > > retain,
> > > > > > > copy, print, distribute or use this message. If you have
> received
> > > > this
> > > > > > > communication in error, please notify the sender and delete all
> > > > copies
> > > > > of
> > > > > > > this message. Accelerite, a Persistent Systems business does
> not
> > > > accept
> > > > > > any
> > > > > > > liability for virus infected mails.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>



-- 
Rafael Weingärtner

Re: Modern template hosting

Posted by Will Stevens <ws...@cloudops.com>.

so this is what I am looking to do.  Please let me know if you have
suggestions for me or think I should be solving the problem a different way.

- We request a new Github repository from the ASF at:
'apache/cloudstack-mirror-list'
- In this repository we track a text file in the 'gh-pages' branch with a
list of valid download mirrors.
- I build a binary to be hosted by the ASF (or at least with the ASF
pointing a domain at the binary and I could potentially host it).  We will
see how they want to handle the hosting of the binary.

The binary would expose a web server which would behave as follows:
- When the 'client' requests a download url the following flow is kicked
off:
-- The mirror list is queried from github (or from a static site hosted on
asf, as we see fit).
-- The Lat/Lon of the 'client' is determined based on their IP.
-- The Lat/Lon for each of the 'mirror's is determined based on an IP
lookup of the hostname.
-- The closest geographical mirror is determined, the target is validated
to be available and the user is redirected.

Some questions I have right now:
- Will every mirror have the same path structure to access the equivalent
resources?
- Should we support adding a path to the mirror url to specify the path to
the base common path?
-- Example: lets say the binary is hosted on 'dl.acs.com' and there are
three mirrors 'abc.com', 'pqr.com/files' and 'xyx.com/downloads'.
-- If the path being requested is '
dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would result in the
following potential paths for the mirrors:
-- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
-- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
-- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'

Does this all make sense?

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <ch...@gmail.com>
wrote:

> My bad. A few lines down, this has been added recently:
>
> this.request.setFollowRedirects(true);
>
> On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <wi...@gmail.com>
> wrote:
>
> > OK. Thanks for the heads up.
> >
> > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <ch...@gmail.com>
> wrote:
> >
> > > Sounds workable. The downloader code in the SSVM won't follow
> redirects I
> > > think.
> > > https://github.com/apache/cloudstack/blob/
> 5511065fc20787619d9cd0444a65a3
> > > 155fc9c921/core/src/com/cloud/storage/template/
> > > HttpTemplateDownloader.java#L93
> > > https://goo.gl/dSi0r5
> > >  Might need to add
> > > client.setRedirectStrategy(new LaxRedirectStrategy());
> > >
> > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <ws...@cloudops.com>
> > > wrote:
> > >
> > > > We haven't opened a ticket yet because we don't have a strategy yet.
> > > >
> > > > What do you guys think of this:
> > > > - We setup a new github repo in the 'apache' org which consists of a
> > > single
> > > > file with a list of active/supported mirrors.
> > > > - I write a small web server, distributed as a binary, which can be
> > > hosted
> > > > by ASF Infra.  This web server will query the current list of mirrors
> > and
> > > > will select one and then do a 302 redirect to that mirror.
> > > >
> > > > The act of 'choosing' a mirror could be done in a number of ways.
> > > > - If we want to define an order, then it could just try from the top
> of
> > > the
> > > > list and work its way down.  It would curl the target to make sure it
> > > gets
> > > > a 200 and if it does, it would do a 302 redirect.
> > > > - Or, if we want to distribute the load across the mirrors, we could
> > pick
> > > > from the list randomly.  Again, doing a curl to verify the mirror is
> up
> > > and
> > > > then doing a redirect.
> > > > - If we want to get fancy, we could do a reverse IP lookup and try to
> > > match
> > > > the requester with their closest geographical mirror.
> > > >
> > > > Thoughts?
> > > >
> > > > *Will STEVENS*
> > > > Lead Developer
> > > >
> > > > <https://goo.gl/NYZ8KK>
> > > >
> > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > chiradeepv@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > What steps are needed to set up a mirror? What does Infra need to
> do?
> > > Has
> > > > > anybody filed a ticket with Infra?
> > > > >
> > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > raja.pullela@accelerite.com>
> > > > > wrote:
> > > > >
> > > > > > Hi will,
> > > > > >
> > > > > > I believe, we didn’t get to close ‘getting a mirror on Apache’
> > > because
> > > > we
> > > > > > needed someone on the Apache Infra side to close this. BTW,
> > > > > > cloudstack-apt.get.eu (I think Nux manages this?) has all/most
> of
> > > the
> > > > > > content.  Once we can close on the Apache mirror for hosting the
> > > > > content, I
> > > > > > can help assist getting the content there.
> > > > > >
> > > > > > For now, we have replicated the download.cloud.com content to ‘
> > > > > > s3.download.accelerite.com’.
> > > > > > Also, we are working on a set of steps/procedure to help with
> this
> > > > > > change.  I will update everyone in about a week’s time on the
> > > details.
> > > > > >
> > > > > > Best,
> > > > > > Raja Pullela
> > > > > > Engineering Team,
> > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > Santa Clara, CA, 95054
> > > > > >
> > > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of
> Will
> > > > > > Stevens" <williamstevens@gmail.com on behalf of
> > > wstevens@cloudops.com>
> > > > > > wrote:
> > > > > >
> > > > > > unfortunately the template mirror conversation got caught up in
> > > details
> > > > > and
> > > > > > nobody took the lead on implementing a solution.
> > > > > >
> > > > > > citrix has been pinging me every couple months to say 'dude, we
> > need
> > > to
> > > > > > remove the dependency on download.citrix.com', but i have not
> had
> > > the
> > > > > > cycles to get in and solve the problem.  the shutdown of that is
> > > > imminent
> > > > > > right now, so we need to solve it asap.
> > > > > >
> > > > > >
> > > > > >
> > > > > > *Will STEVENS*
> > > > > > Lead Developer
> > > > > >
> > > > > > <https://goo.gl/NYZ8KK>
> > > > > >
> > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > paul.angus@shapeblue.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Nathan,
> > > > > > >
> > > > > > > Ideally, if you put the template location in (or use a template
> > > > defined
> > > > > > > in)  test_data.py then the actual location can be overridden by
> > > > anyone
> > > > > > > testing.
> > > > > > >
> > > > > > > For Trillian, we've copied all of the templates that people
> have
> > > > define
> > > > > > to
> > > > > > > a local repo and then replace the URLs in test_data.py to
> reduce
> > > > > > bandwidth
> > > > > > > use and download times.
> > > > > > >
> > > > > > > Ie:
> > > > > > >
> > > > > > >             "bootableIso":
> > > > > > >                 {
> > > > > > >                     "displaytext": "Test Bootable ISO",
> > > > > > >                     "name": "testISO",
> > > > > > >                     "bootable": True,
> > > > > > >                     "ispublic": False,
> > > > > > >                     "url": "{{ marvin_images_location
> > > > > > > }}/TinyCore-current.iso",
> > > > > > >                     "ostype": 'Other Linux (64-bit)',
> > > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > > >         },
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I thought that we had come up with a solution for
> > > download.cloud.com
> > > > ,
> > > > > by
> > > > > > > having a mirrorlist hosted in Community Apache 'space' with
> > anyone
> > > > able
> > > > > > to
> > > > > > > out themselves forward as a mirror.
> > > > > > > But I must admit I lost track of whether anyone made the
> > requisite
> > > > > > changes
> > > > > > > in code....
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Kind regards,
> > > > > > >
> > > > > > > Paul Angus
> > > > > > >
> > > > > > > paul.angus@shapeblue.com
> > > > > > > www.shapeblue.com
> > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > @shapeblue
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: williamstevens@gmail.com [mailto:williamstevens@gmail.
> com]
> > > On
> > > > > > > Behalf Of Will Stevens
> > > > > > > Sent: 24 February 2017 16:30
> > > > > > > To: dev@cloudstack.apache.org
> > > > > > > Subject: Re: Modern template hosting
> > > > > > >
> > > > > > > this is a hard questions.  in general, we should be setting up
> a
> > > > mirror
> > > > > > on
> > > > > > > some cloudstack/apache domain and then mirror to other provided
> > > > > > templates.
> > > > > > >
> > > > > > > we MUST come up with a solution to deprecate '
> download.cloud.com
> > ',
> > > > > that
> > > > > > > is going to be going away any day now.
> > > > > > >
> > > > > > > i don't know the right way to solve this to be honest, but if
> you
> > > > have
> > > > > > > ideas, i am willing to help.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *Will STEVENS*
> > > > > > > Lead Developer
> > > > > > >
> > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > >
> > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> > njohnson@ena.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > So not to re-open a can of worms, but I’m in a situation
> where
> > I
> > > > need
> > > > > > > > to come up with a Marvin component test that depends on a
> > > template
> > > > > > > > based on a kernel that’s relatively new, i.e., newer than
> > Centos
> > > > 5.3
> > > > > /
> > > > > > > Ubuntu 10.04 .
> > > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for
> > KVM),
> > > > but
> > > > > > > > from looking at the thread "Migrating CloudStack content from
> > > > > > > > download.cloud.com” it looks like there is resistance to
> using
> > > > this
> > > > > at
> > > > > > > > least for hosting system vm templates over concerns of
> > > neutrality.
> > > > > > > > Would this be suitable for a component test?  If not, what
> is a
> > > > > > “blessed”
> > > > > > > > template location?
> > > > > > > >
> > > > > > > > Thanks in advance!
> > > > > > > >
> > > > > > > > Nathan Johnson
> > > > > > > > R&D Engineer
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > Nashville, TN 37211
> > > > > > > > General Office: 615-312-6000
> > > > > > > >
> > > > > > > > website | blog | support
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > DISCLAIMER
> > > > > > ==========
> > > > > > This e-mail may contain privileged and confidential information
> > which
> > > > is
> > > > > > the property of Accelerite, a Persistent Systems business. It is
> > > > intended
> > > > > > only for the use of the individual or entity to which it is
> > > addressed.
> > > > If
> > > > > > you are not the intended recipient, you are not authorized to
> read,
> > > > > retain,
> > > > > > copy, print, distribute or use this message. If you have received
> > > this
> > > > > > communication in error, please notify the sender and delete all
> > > copies
> > > > of
> > > > > > this message. Accelerite, a Persistent Systems business does not
> > > accept
> > > > > any
> > > > > > liability for virus infected mails.
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Modern template hosting

Posted by Chiradeep Vittal <ch...@gmail.com>.

My bad. A few lines down, this has been added recently:

this.request.setFollowRedirects(true);

On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <wi...@gmail.com>
wrote:

> OK. Thanks for the heads up.
>
> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:
>
> > Sounds workable. The downloader code in the SSVM won't follow redirects I
> > think.
> > https://github.com/apache/cloudstack/blob/5511065fc20787619d9cd0444a65a3
> > 155fc9c921/core/src/com/cloud/storage/template/
> > HttpTemplateDownloader.java#L93
> > https://goo.gl/dSi0r5
> >  Might need to add
> > client.setRedirectStrategy(new LaxRedirectStrategy());
> >
> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <ws...@cloudops.com>
> > wrote:
> >
> > > We haven't opened a ticket yet because we don't have a strategy yet.
> > >
> > > What do you guys think of this:
> > > - We setup a new github repo in the 'apache' org which consists of a
> > single
> > > file with a list of active/supported mirrors.
> > > - I write a small web server, distributed as a binary, which can be
> > hosted
> > > by ASF Infra.  This web server will query the current list of mirrors
> and
> > > will select one and then do a 302 redirect to that mirror.
> > >
> > > The act of 'choosing' a mirror could be done in a number of ways.
> > > - If we want to define an order, then it could just try from the top of
> > the
> > > list and work its way down.  It would curl the target to make sure it
> > gets
> > > a 200 and if it does, it would do a 302 redirect.
> > > - Or, if we want to distribute the load across the mirrors, we could
> pick
> > > from the list randomly.  Again, doing a curl to verify the mirror is up
> > and
> > > then doing a redirect.
> > > - If we want to get fancy, we could do a reverse IP lookup and try to
> > match
> > > the requester with their closest geographical mirror.
> > >
> > > Thoughts?
> > >
> > > *Will STEVENS*
> > > Lead Developer
> > >
> > > <https://goo.gl/NYZ8KK>
> > >
> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> chiradeepv@gmail.com
> > >
> > > wrote:
> > >
> > > > What steps are needed to set up a mirror? What does Infra need to do?
> > Has
> > > > anybody filed a ticket with Infra?
> > > >
> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > raja.pullela@accelerite.com>
> > > > wrote:
> > > >
> > > > > Hi will,
> > > > >
> > > > > I believe, we didn’t get to close ‘getting a mirror on Apache’
> > because
> > > we
> > > > > needed someone on the Apache Infra side to close this. BTW,
> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has all/most of
> > the
> > > > > content.  Once we can close on the Apache mirror for hosting the
> > > > content, I
> > > > > can help assist getting the content there.
> > > > >
> > > > > For now, we have replicated the download.cloud.com content to ‘
> > > > > s3.download.accelerite.com’.
> > > > > Also, we are working on a set of steps/procedure to help with this
> > > > > change.  I will update everyone in about a week’s time on the
> > details.
> > > > >
> > > > > Best,
> > > > > Raja Pullela
> > > > > Engineering Team,
> > > > > Accelerite, 2055 Laurelwood Road,
> > > > > Santa Clara, CA, 95054
> > > > >
> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of Will
> > > > > Stevens" <williamstevens@gmail.com on behalf of
> > wstevens@cloudops.com>
> > > > > wrote:
> > > > >
> > > > > unfortunately the template mirror conversation got caught up in
> > details
> > > > and
> > > > > nobody took the lead on implementing a solution.
> > > > >
> > > > > citrix has been pinging me every couple months to say 'dude, we
> need
> > to
> > > > > remove the dependency on download.citrix.com', but i have not had
> > the
> > > > > cycles to get in and solve the problem.  the shutdown of that is
> > > imminent
> > > > > right now, so we need to solve it asap.
> > > > >
> > > > >
> > > > >
> > > > > *Will STEVENS*
> > > > > Lead Developer
> > > > >
> > > > > <https://goo.gl/NYZ8KK>
> > > > >
> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > paul.angus@shapeblue.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Nathan,
> > > > > >
> > > > > > Ideally, if you put the template location in (or use a template
> > > defined
> > > > > > in)  test_data.py then the actual location can be overridden by
> > > anyone
> > > > > > testing.
> > > > > >
> > > > > > For Trillian, we've copied all of the templates that people have
> > > define
> > > > > to
> > > > > > a local repo and then replace the URLs in test_data.py to reduce
> > > > > bandwidth
> > > > > > use and download times.
> > > > > >
> > > > > > Ie:
> > > > > >
> > > > > >             "bootableIso":
> > > > > >                 {
> > > > > >                     "displaytext": "Test Bootable ISO",
> > > > > >                     "name": "testISO",
> > > > > >                     "bootable": True,
> > > > > >                     "ispublic": False,
> > > > > >                     "url": "{{ marvin_images_location
> > > > > > }}/TinyCore-current.iso",
> > > > > >                     "ostype": 'Other Linux (64-bit)',
> > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > >         },
> > > > > >
> > > > > >
> > > > > >
> > > > > > I thought that we had come up with a solution for
> > download.cloud.com
> > > ,
> > > > by
> > > > > > having a mirrorlist hosted in Community Apache 'space' with
> anyone
> > > able
> > > > > to
> > > > > > out themselves forward as a mirror.
> > > > > > But I must admit I lost track of whether anyone made the
> requisite
> > > > > changes
> > > > > > in code....
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Kind regards,
> > > > > >
> > > > > > Paul Angus
> > > > > >
> > > > > > paul.angus@shapeblue.com
> > > > > > www.shapeblue.com
> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > @shapeblue
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: williamstevens@gmail.com [mailto:williamstevens@gmail.com]
> > On
> > > > > > Behalf Of Will Stevens
> > > > > > Sent: 24 February 2017 16:30
> > > > > > To: dev@cloudstack.apache.org
> > > > > > Subject: Re: Modern template hosting
> > > > > >
> > > > > > this is a hard questions.  in general, we should be setting up a
> > > mirror
> > > > > on
> > > > > > some cloudstack/apache domain and then mirror to other provided
> > > > > templates.
> > > > > >
> > > > > > we MUST come up with a solution to deprecate 'download.cloud.com
> ',
> > > > that
> > > > > > is going to be going away any day now.
> > > > > >
> > > > > > i don't know the right way to solve this to be honest, but if you
> > > have
> > > > > > ideas, i am willing to help.
> > > > > >
> > > > > >
> > > > > >
> > > > > > *Will STEVENS*
> > > > > > Lead Developer
> > > > > >
> > > > > > <https://goo.gl/NYZ8KK>
> > > > > >
> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> njohnson@ena.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > So not to re-open a can of worms, but I’m in a situation where
> I
> > > need
> > > > > > > to come up with a Marvin component test that depends on a
> > template
> > > > > > > based on a kernel that’s relatively new, i.e., newer than
> Centos
> > > 5.3
> > > > /
> > > > > > Ubuntu 10.04 .
> > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for
> KVM),
> > > but
> > > > > > > from looking at the thread "Migrating CloudStack content from
> > > > > > > download.cloud.com” it looks like there is resistance to using
> > > this
> > > > at
> > > > > > > least for hosting system vm templates over concerns of
> > neutrality.
> > > > > > > Would this be suitable for a component test?  If not, what is a
> > > > > “blessed”
> > > > > > > template location?
> > > > > > >
> > > > > > > Thanks in advance!
> > > > > > >
> > > > > > > Nathan Johnson
> > > > > > > R&D Engineer
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > Nashville, TN 37211
> > > > > > > General Office: 615-312-6000
> > > > > > >
> > > > > > > website | blog | support
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > DISCLAIMER
> > > > > ==========
> > > > > This e-mail may contain privileged and confidential information
> which
> > > is
> > > > > the property of Accelerite, a Persistent Systems business. It is
> > > intended
> > > > > only for the use of the individual or entity to which it is
> > addressed.
> > > If
> > > > > you are not the intended recipient, you are not authorized to read,
> > > > retain,
> > > > > copy, print, distribute or use this message. If you have received
> > this
> > > > > communication in error, please notify the sender and delete all
> > copies
> > > of
> > > > > this message. Accelerite, a Persistent Systems business does not
> > accept
> > > > any
> > > > > liability for virus infected mails.
> > > > >
> > > >
> > >
> >
>

Re: Modern template hosting

Posted by Rafael Weingärtner <ra...@gmail.com>.

I liked this proposal. I would only question if the ASF Infra would expose
directly the application, or if it would put the application behind a web
server. If we have a web server in front of it, it gets easier to balance
the “proxy” mechanism between multiple instances of this application. I do
not know how is the Infra systems there (do they have a private cloud using
ACS?!), but I would assume they could configure a scale-out mechanism to
replicate instances as soon as the current ones get saturated.

On Fri, Mar 3, 2017 at 12:54 PM, Will Stevens <sw...@apache.org> wrote:

> So now I am leaning towards the following implementation.  Use a query
> parameter (more accessible than a header) to choose between delivery
> options with a sane default if no query param is added.
>
> The following (potential) options would be available:
> a) [ default, no params ] current implementation.  first mirror to answer
> to the web server would handle the request.
> b) [ ?serve=mirrors ] return a 300 with a list of 'valid' mirrors (head
> check on resource path).
> c) [ ?serve=closest ] the closest 'valid' geographic mirror serves the
> request.  falls back to (a) on error.
> d) [ ?serve=proxy ] the web server would use (a) and stream the
> corresponding mirror back to the client.
>
> We would need a documented recommendation for users based on version
> number.  The true legacy would have to use (d). It defaults to a sane
> default which can handle the slow down of a mirror.  It also lays the
> foundation for optimizing (d) if we need to manage the load across the
> mirrors in the case of proxying.
>
> Thoughts?  I feel like we are getting to a point where we understand what
> our potential implementation paths are.
>
> *Will Stevens*
>
> PS - If you see this, you should consider coming to CCC in Miami.  Travel
> assistance closes on March 8th!!!
>
>
>
>
> On Fri, Mar 3, 2017 at 12:31 PM, Will Stevens <ws...@cloudops.com>
> wrote:
>
> > 1) If the legacy implementations do not support redirects, that does
> cause
> > a problem.  A potential solution in that case is to have the web server
> > actually proxy the download, but that is not ideal and I would like to
> > avoid it if possible.  Thanks for bringing that up Chiradeep.
> >
> > 2) I think we need to have a single URL which people can target.  Once
> > they make the switch to the new URL, we want the implementation to be
> able
> > to handle mirror failures without affecting the end client.  We want to
> > avoid the situation where an ACS user will ever have to change this URL
> > more than once.  Mirror failures SHOULD NOT affect the ACS users assuming
> > there is still at least one mirror who can serve the requested resource.
> >
> > These are obviously my personal opinions and others will probably have
> > differing opinions.
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal <ch...@gmail.com>
> > wrote:
> >
> >> 1. If you are targeting legacy installations, they are not able to
> follow
> >> redirects. The line of code that added this capability was added on
> 11/16.
> >> 2. If you trust the users to edit the database to change the URL, you
> can
> >> trust them to change it to anything. Just document a known list of good
> >> template locations.
> >>
> >> On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <ws...@cloudops.com>
> >> wrote:
> >>
> >> > So the main issue I see with this is the following.
> >> >
> >> > This implementation is designed to target legacy installations which
> >> will
> >> > be affected when download.cloud.com disappears.  These people will
> >> need to
> >> > make a DB change to replace the 'download.cloud.com' with some other
> >> url
> >> > (maybe 'download.cloudstack.org').  Once the DB has been updated, we
> >> can
> >> > not expect anything else of the client, they need to be able to just
> >> > continue operation as they were without any need to rebuild or upgrade
> >> > their ACS.
> >> >
> >> > If we try to force the decision to the client, then we break the
> legacy
> >> > implementations.  Since the SSVM is likely going to be the client in
> >> some
> >> > cases, and since it does not already have the logic to handle the 300
> >> > approach correctly, I feel like it is not a viable solution for the
> >> legacy
> >> > deployments.
> >> >
> >> > In addition to that.  I don't think the client is in any better
> >> position to
> >> > make the mirror decision than I am.  I am likely able to make a more
> >> > 'educated' decision than the client would be able to because I can do
> >> > different tests on the endpoint(s) before making a final decision.
> >> >
> >> > Does this make sense or am I missing something obvious?
> >> >
> >> > *Will STEVENS*
> >> > Lead Developer
> >> >
> >> > <https://goo.gl/NYZ8KK>
> >> >
> >> > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
> >> > rafaelweingartner@gmail.com> wrote:
> >> >
> >> > > I think I understood you, but I did not understand you.
> >> > >
> >> > > Let me see if I can create an example to illustrate how we could do
> >> using
> >> > > the HTTP 300 code.
> >> > >
> >> > > The HTTP 300 code indicates clients that he/she/it has multiple
> >> choices
> >> > to
> >> > > access the requested resource. If the request is a HEAD, this is the
> >> only
> >> > > thing that the server returns. If the request if a GET, then the
> >> server
> >> > > returns the choices for the client. The response type is defined on
> >> the
> >> > > server based on the Content-Type, user agent and others.
> >> > >
> >> > > Let’s say the server receives a GET request and informs the
> >> content-type
> >> > as
> >> > > JSON (application/JSON). Your application could get the mirrors
> list,
> >> do
> >> > > the appending process for the requested resource path, and then
> >> return a
> >> > > JSON array with the possible mirror the client can use. Then, the
> >> client
> >> > > gets this list and does the processing required to select a mirror.
> >> > >
> >> > > Now let’s say that a human access the link using a browser. Then,
> the
> >> > > server should receive something like (text/plain) as the
> >> content-type; I
> >> > > think depending on the browser this field may differ a little (it
> >> would
> >> > > require some checking). Anyways, the server detects that it is a
> >> “human”
> >> > > requesting the resource, then we could serve an HTML page with a
> >> > Javascript
> >> > > that uses the mirror list. This Javascript could do some checking
> and
> >> > > choose the best mirror for that specific.
> >> > >
> >> > > Why do I think this approach is interesting?
> >> > > The application you developed would be used only to retrieve valid
> and
> >> > > trusted sources of system VM images (a trusted repository for
> mirrors
> >> > that
> >> > > we as a community vouch for).  Also, we shift the decision process
> >> > > regarding mirrors from the server to the client. Then, it is up to
> >> > clients
> >> > > to select mirrors, and not up to us (or some of our
> implementations).
> >> > >
> >> > > Did this help?
> >> > > BTW: I am not saying I am against the way you proposed, which would
> >> work
> >> > > fine. It is merely a suggestion using a different perspective.
> >> > >
> >> > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
> >> williamstevens@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > @rafael: in general, I don't think the client should ever be
> making
> >> a
> >> > > > choice about a mirror. We have to assume we are working with a
> >> scripted
> >> > > > application and anything hard coded in that implementation on the
> >> > client
> >> > > > side is a risk.
> >> > > >
> >> > > > I may not be understanding the use of the 300 approach, so I will
> >> > > research
> >> > > > it to see if I can make it fit.
> >> > > >
> >> > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
> >> > > rafaelweingartner@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > Will great job.
> >> > > >
> >> > > > I had the same doubt as Daan.
> >> > > >
> >> > > > Have you considered using HTTP 300 working mode? Then, we could
> let
> >> the
> >> > > > client decide which mirror is the best(closest?). Section “10.3.1”
> >> of
> >> > [1]
> >> > > > talks about it; this would put the pressure on deciding upon a
> >> mirror
> >> > on
> >> > > > the client side, but I think it would be fair.
> >> > > >
> >> > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
> >> > > >
> >> > > > PS: You really like very short variables names!
> >> > > >
> >> > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
> >> > > daan.hoogland@shapeblue.com
> >> > > > >
> >> > > > wrote:
> >> > > >
> >> > > > > Nice little thing Will,
> >> > > > >
> >> > > > > One question: if I read the code correctly it ‘go’es and tries
> all
> >> > > > mirrors
> >> > > > > at once and whichever responses first is redirected to the
> client.
> >> > This
> >> > > > > might well be the same every time. This might be the one closest
> >> to
> >> > the
> >> > > > > mirror302 and not to the client. Is that correct or did I miss a
> >> > > weighing
> >> > > > > algorithm hidden in there?
> >> > > > >
> >> > > > > Good coding,
> >> > > > >
> >> > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
> >> > > > >
> >> > > > >     Hey All,
> >> > > > >     Please review this repo and tell me what you think:
> >> > > > >     https://github.com/swill/mirror302
> >> > > > >
> >> > > > >     Let me know if you have questions or you would like me to
> make
> >> > > > changes
> >> > > > >     before I bring this topic up with ASF Infra and open a
> ticket
> >> to
> >> > > get
> >> > > > > this
> >> > > > >     implemented.
> >> > > > >
> >> > > > >     Cheers,
> >> > > > >
> >> > > > >     *Will Stevens*
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> >> > > wstevens@cloudops.com
> >> > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > >     > I am building a short term solution right now.  I
> hopefully
> >> > will
> >> > > > > have the
> >> > > > >     > building blocks in place by the weekend so we can start
> >> working
> >> > > > with
> >> > > > > Infra
> >> > > > >     > to get it in place.
> >> > > > >     >
> >> > > > >     > We will have a web server which we can point something
> like
> >> > > > >     > downloads.cloudstack.org (or whatever url) at.  Then we
> >> will
> >> > > have
> >> > > > > an ASF
> >> > > > >     > repo which tracks a mirror list and exposes a static site
> >> to be
> >> > > > able
> >> > > > > to
> >> > > > >     > query the mirror list.
> >> > > > >     >
> >> > > > >     > The web server will receive a request and will do a 302
> >> > redirect
> >> > > to
> >> > > > > the
> >> > > > >     > appropriate resource on one of the mirrors.
> >> > > > >     >
> >> > > > >     > This gets us started.  Legacy environments will have to do
> >> a DB
> >> > > > > change to
> >> > > > >     > change from 'download.cloud.com' to '
> >> downloads.cloudstack.org'
> >> > > (or
> >> > > > >     > whatever), but otherwise it should be pretty simple.
> >> > > > >     >
> >> > > > >     > We can improve how we deliver templates going forward, but
> >> this
> >> > > > > obviously
> >> > > > >     > requires some discussion still.  I will try to buy us some
> >> time
> >> > > > with
> >> > > > > an
> >> > > > >     > implementation which solves our problems today...
> >> > > > >     >
> >> > > > >     > *Will STEVENS*
> >> > > > >     > Lead Developer
> >> > > > >     >
> >> > > > >     > <https://goo.gl/NYZ8KK>
> >> > > > >     >
> >> > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> >> > > > >     > abhinandan.prateek@shapeblue.com> wrote:
> >> > > > >     >
> >> > > > >     >>
> >> > > > >     >> Initial seeding is a manual step and that is not going to
> >> > change
> >> > > > in
> >> > > > > near
> >> > > > >     >> future. A handy list of official places from where these
> >> > > templates
> >> > > > > can be
> >> > > > >     >> downloaded won’t harm this part of the installation. Just
> >> > allow
> >> > > > > anyone
> >> > > > >     >> credible who follows democratic process to publish their
> >> > > templates
> >> > > > > with
> >> > > > >     >> apache’s blessings.
> >> > > > >     >>
> >> > > > >     >>
> >> > > > >     >> Coming to urls hard coded in db, the popular suggestion
> is
> >> to
> >> > > > point
> >> > > > > them
> >> > > > >     >> to mirrors. This looks good but this is going to take
> some
> >> > > effort
> >> > > > > that has
> >> > > > >     >> not materialised yet. Once this change is made it will be
> >> > going
> >> > > in
> >> > > > > some
> >> > > > >     >> future release like 4.11; the problem still remains for
> the
> >> > > > > releases that
> >> > > > >     >> are already out there. In case access to
> >> download.cloud.com
> >> > is
> >> > > > > dropped
> >> > > > >     >> at some point we will have to document the procedure such
> >> that
> >> > > > > people know
> >> > > > >     >> how to make these stable releases work. That is where the
> >> > > > procedure
> >> > > > >     >> documented here https://shankerbalan.net/blog/
> >> > > > >     >> seed-cloudstack-templates-offline/ should be officially
> >> > adapted
> >> > > > > with a
> >> > > > >     >> marketplace to pick up templates of choosing. Maybe we
> can
> >> > have
> >> > > a
> >> > > > > pre-setup
> >> > > > >     >> script to make choices instead of written wiki. Note that
> >> this
> >> > > > will
> >> > > > > be
> >> > > > >     >> anyway needed by the older releases.
> >> > > > >     >>
> >> > > > >     >>
> >> > > > >     >>
> >> > > > >     >>
> >> > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
> >> paul.angus@shapeblue.com>
> >> > > > > wrote:
> >> > > > >     >>
> >> > > > >     >> >-1
> >> > > > >     >> >
> >> > > > >     >> >We are trying to get MORE people to use CloudStack.  And
> >> > > raising
> >> > > > > the
> >> > > > >     >> barrier to entry does nothing to help that.
> >> > > > >     >> >
> >> > > > >     >> >
> >> > > > >     >> >Kind regards,
> >> > > > >     >> >
> >> > > > >     >> >Paul Angus
> >> > > > >     >> >
> >> > > > >     >> >paul.angus@shapeblue.com
> >> > > > >     >> >www.shapeblue.com
> >> > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >> > > > >     >> >@shapeblue
> >> > > > >     >> >
> >> > > > >     >> >
> >> > > > >     >> >
> >> > > > >     >> >
> >> > > > >     >> >-----Original Message-----
> >> > > > >     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@
> >> > > > shapeblue.com]
> >> > > > >     >> >Sent: 01 March 2017 09:39
> >> > > > >     >> >To: dev@cloudstack.apache.org
> >> > > > >     >> >Subject: Re: Modern template hosting
> >> > > > >     >> >
> >> > > > >     >> >+1
> >> > > > >     >> >
> >> > > > >     >> >After seeing this discussion running in circles several
> >> > times,
> >> > > I
> >> > > > > think
> >> > > > >     >> we should at least get started with the simplest option.
> >> > > > >     >> >
> >> > > > >     >> >
> >> > > > >     >> >
> >> > > > >     >> >
> >> > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
> >> > > chiradeepv@gmail.com>
> >> > > > > wrote:
> >> > > > >     >> >
> >> > > > >     >> >>The alternative foolproof way is to simply not provide
> >> the
> >> > > > > automatic
> >> > > > >     >> >>download.
> >> > > > >     >> >>
> >> > > > >     >> >>Just document the mirror list URL where the templates
> >> can be
> >> > > > > found. I
> >> > > > >     >> >>don't know why people reject this option.
> >> > > > >     >> >>
> >> > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
> >> > > > templates-offline/
> >> > > > >     >> >>
> >> > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> >> > > > >     >> >><wi...@gmail.com>
> >> > > > >     >> >>wrote:
> >> > > > >     >> >>
> >> > > > >     >> >>> OK. Thanks for the heads up.
> >> > > > >     >> >>>
> >> > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> >> > > > > chiradeepv@gmail.com>
> >> > > > >     >> wrote:
> >> > > > >     >> >>>
> >> > > > >     >> >>> > Sounds workable. The downloader code in the SSVM
> >> won't
> >> > > > follow
> >> > > > >     >> >>> > redirects I think.
> >> > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
> >> > > > 5511065fc20787619d
> >> > > > > 9cd0444
> >> > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/
> >> storage/template/
> >> > > > >     >> >>> > HttpTemplateDownloader.java#L93
> >> > > > >     >> >>> > https://goo.gl/dSi0r5
> >> > > > >     >> >>> >  Might need to add
> >> > > > >     >> >>> > client.setRedirectStrategy(new
> >> LaxRedirectStrategy());
> >> > > > >     >> >>> >
> >> > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> >> > > > >     >> >>> > <ws...@cloudops.com>
> >> > > > >     >> >>> > wrote:
> >> > > > >     >> >>> >
> >> > > > >     >> >>> > > We haven't opened a ticket yet because we don't
> >> have a
> >> > > > > strategy
> >> > > > >     >> yet.
> >> > > > >     >> >>> > >
> >> > > > >     >> >>> > > What do you guys think of this:
> >> > > > >     >> >>> > > - We setup a new github repo in the 'apache' org
> >> which
> >> > > > > consists
> >> > > > >     >> >>> > > of a
> >> > > > >     >> >>> > single
> >> > > > >     >> >>> > > file with a list of active/supported mirrors.
> >> > > > >     >> >>> > > - I write a small web server, distributed as a
> >> binary,
> >> > > > > which can
> >> > > > >     >> >>> > > be
> >> > > > >     >> >>> > hosted
> >> > > > >     >> >>> > > by ASF Infra.  This web server will query the
> >> current
> >> > > list
> >> > > > > of
> >> > > > >     >> >>> > > mirrors
> >> > > > >     >> >>> and
> >> > > > >     >> >>> > > will select one and then do a 302 redirect to
> that
> >> > > mirror.
> >> > > > >     >> >>> > >
> >> > > > >     >> >>> > > The act of 'choosing' a mirror could be done in a
> >> > number
> >> > > > of
> >> > > > > ways.
> >> > > > >     >> >>> > > - If we want to define an order, then it could
> just
> >> > try
> >> > > > > from the
> >> > > > >     >> >>> > > top of
> >> > > > >     >> >>> > the
> >> > > > >     >> >>> > > list and work its way down.  It would curl the
> >> target
> >> > to
> >> > > > > make
> >> > > > >     >> >>> > > sure it
> >> > > > >     >> >>> > gets
> >> > > > >     >> >>> > > a 200 and if it does, it would do a 302 redirect.
> >> > > > >     >> >>> > > - Or, if we want to distribute the load across
> the
> >> > > > mirrors,
> >> > > > > we
> >> > > > >     >> >>> > > could
> >> > > > >     >> >>> pick
> >> > > > >     >> >>> > > from the list randomly.  Again, doing a curl to
> >> verify
> >> > > the
> >> > > > > mirror
> >> > > > >     >> >>> > > is up
> >> > > > >     >> >>> > and
> >> > > > >     >> >>> > > then doing a redirect.
> >> > > > >     >> >>> > > - If we want to get fancy, we could do a reverse
> IP
> >> > > lookup
> >> > > > > and
> >> > > > >     >> >>> > > try to
> >> > > > >     >> >>> > match
> >> > > > >     >> >>> > > the requester with their closest geographical
> >> mirror.
> >> > > > >     >> >>> > >
> >> > > > >     >> >>> > > Thoughts?
> >> > > > >     >> >>> > >
> >> > > > >     >> >>> > > *Will STEVENS*
> >> > > > >     >> >>> > > Lead Developer
> >> > > > >     >> >>> > >
> >> > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
> >> > > > >     >> >>> > >
> >> > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep
> Vittal
> >> <
> >> > > > >     >> >>> chiradeepv@gmail.com
> >> > > > >     >> >>> > >
> >> > > > >     >> >>> > > wrote:
> >> > > > >     >> >>> > >
> >> > > > >     >> >>> > > > What steps are needed to set up a mirror? What
> >> does
> >> > > > Infra
> >> > > > > need
> >> > > > >     >> to do?
> >> > > > >     >> >>> > Has
> >> > > > >     >> >>> > > > anybody filed a ticket with Infra?
> >> > > > >     >> >>> > > >
> >> > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela
> <
> >> > > > >     >> >>> > > > raja.pullela@accelerite.com>
> >> > > > >     >> >>> > > > wrote:
> >> > > > >     >> >>> > > >
> >> > > > >     >> >>> > > > > Hi will,
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > > > I believe, we didn’t get to close ‘getting a
> >> > mirror
> >> > > on
> >> > > > > Apache’
> >> > > > >     >> >>> > because
> >> > > > >     >> >>> > > we
> >> > > > >     >> >>> > > > > needed someone on the Apache Infra side to
> >> close
> >> > > this.
> >> > > > > BTW,
> >> > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages
> >> this?)
> >> > > has
> >> > > > >     >> >>> > > > > all/most of
> >> > > > >     >> >>> > the
> >> > > > >     >> >>> > > > > content.  Once we can close on the Apache
> >> mirror
> >> > for
> >> > > > > hosting
> >> > > > >     >> >>> > > > > the
> >> > > > >     >> >>> > > > content, I
> >> > > > >     >> >>> > > > > can help assist getting the content there.
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > > > For now, we have replicated the
> >> > download.cloud.com
> >> > > > > content to
> >> > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
> >> > > > >     >> >>> > > > > Also, we are working on a set of
> >> steps/procedure
> >> > to
> >> > > > > help with
> >> > > > >     >> >>> > > > > this change.  I will update everyone in
> about a
> >> > > week’s
> >> > > > > time
> >> > > > >     >> >>> > > > > on the
> >> > > > >     >> >>> > details.
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > > > Best,
> >> > > > >     >> >>> > > > > Raja Pullela
> >> > > > >     >> >>> > > > > Engineering Team,
> >> > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa
> Clara,
> >> CA,
> >> > > > 95054
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "
> >> williamstevens@gmail.com
> >> > on
> >> > > > > behalf of
> >> > > > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com on
> >> behalf
> >> > > of
> >> > > > >     >> >>> > wstevens@cloudops.com>
> >> > > > >     >> >>> > > > > wrote:
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > > > unfortunately the template mirror
> conversation
> >> got
> >> > > > > caught up
> >> > > > >     >> >>> > > > > in
> >> > > > >     >> >>> > details
> >> > > > >     >> >>> > > > and
> >> > > > >     >> >>> > > > > nobody took the lead on implementing a
> >> solution.
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > > > citrix has been pinging me every couple
> months
> >> to
> >> > > say
> >> > > > > 'dude,
> >> > > > >     >> >>> > > > > we
> >> > > > >     >> >>> need
> >> > > > >     >> >>> > to
> >> > > > >     >> >>> > > > > remove the dependency on download.citrix.com
> ',
> >> > but
> >> > > i
> >> > > > > have not
> >> > > > >     >> >>> > > > > had
> >> > > > >     >> >>> > the
> >> > > > >     >> >>> > > > > cycles to get in and solve the problem.  the
> >> > > shutdown
> >> > > > > of that
> >> > > > >     >> >>> > > > > is
> >> > > > >     >> >>> > > imminent
> >> > > > >     >> >>> > > > > right now, so we need to solve it asap.
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > > > *Will STEVENS*
> >> > > > >     >> >>> > > > > Lead Developer
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus
> <
> >> > > > >     >> >>> > paul.angus@shapeblue.com
> >> > > > >     >> >>> > > >
> >> > > > >     >> >>> > > > > wrote:
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > > > > Hi Nathan,
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > > Ideally, if you put the template location
> in
> >> (or
> >> > > use
> >> > > > a
> >> > > > >     >> >>> > > > > > template
> >> > > > >     >> >>> > > defined
> >> > > > >     >> >>> > > > > > in)  test_data.py then the actual location
> >> can
> >> > be
> >> > > > >     >> >>> > > > > > overridden by
> >> > > > >     >> >>> > > anyone
> >> > > > >     >> >>> > > > > > testing.
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > > For Trillian, we've copied all of the
> >> templates
> >> > > that
> >> > > > > people
> >> > > > >     >> >>> > > > > > have
> >> > > > >     >> >>> > > define
> >> > > > >     >> >>> > > > > to
> >> > > > >     >> >>> > > > > > a local repo and then replace the URLs in
> >> > > > > test_data.py to
> >> > > > >     >> >>> > > > > > reduce
> >> > > > >     >> >>> > > > > bandwidth
> >> > > > >     >> >>> > > > > > use and download times.
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > > Ie:
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > >             "bootableIso":
> >> > > > >     >> >>> > > > > >                 {
> >> > > > >     >> >>> > > > > >                     "displaytext": "Test
> >> > Bootable
> >> > > > > ISO",
> >> > > > >     >> >>> > > > > >                     "name": "testISO",
> >> > > > >     >> >>> > > > > >                     "bootable": True,
> >> > > > >     >> >>> > > > > >                     "ispublic": False,
> >> > > > >     >> >>> > > > > >                     "url": "{{
> >> > > > marvin_images_location
> >> > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
> >> > > > >     >> >>> > > > > >                     "ostype": 'Other Linux
> >> > > > (64-bit)',
> >> > > > >     >> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
> >> > > > >     >> >>> > > > > >         },
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > > I thought that we had come up with a
> solution
> >> > for
> >> > > > >     >> >>> > download.cloud.com
> >> > > > >     >> >>> > > ,
> >> > > > >     >> >>> > > > by
> >> > > > >     >> >>> > > > > > having a mirrorlist hosted in Community
> >> Apache
> >> > > > > 'space' with
> >> > > > >     >> >>> anyone
> >> > > > >     >> >>> > > able
> >> > > > >     >> >>> > > > > to
> >> > > > >     >> >>> > > > > > out themselves forward as a mirror.
> >> > > > >     >> >>> > > > > > But I must admit I lost track of whether
> >> anyone
> >> > > made
> >> > > > > the
> >> > > > >     >> >>> requisite
> >> > > > >     >> >>> > > > > changes
> >> > > > >     >> >>> > > > > > in code....
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > > Kind regards,
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > > Paul Angus
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > > paul.angus@shapeblue.com
> >> > > > >     >> >>> > > > > > www.shapeblue.com
> >> > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden, London
> WC2N
> >> > > 4HSUK
> >> > > > >     >> >>> > > > > > @shapeblue
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > > -----Original Message-----
> >> > > > >     >> >>> > > > > > From: williamstevens@gmail.com
> >> > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
> >> > > > >     >> >>> > On
> >> > > > >     >> >>> > > > > > Behalf Of Will Stevens
> >> > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
> >> > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
> >> > > > >     >> >>> > > > > > Subject: Re: Modern template hosting
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > > this is a hard questions.  in general, we
> >> should
> >> > > be
> >> > > > > setting
> >> > > > >     >> >>> > > > > > up a
> >> > > > >     >> >>> > > mirror
> >> > > > >     >> >>> > > > > on
> >> > > > >     >> >>> > > > > > some cloudstack/apache domain and then
> >> mirror to
> >> > > > other
> >> > > > >     >> >>> > > > > > provided
> >> > > > >     >> >>> > > > > templates.
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > > we MUST come up with a solution to
> deprecate
> >> > > > >     >> >>> > > > > > 'download.cloud.com
> >> > > > >     >> >>> ',
> >> > > > >     >> >>> > > > that
> >> > > > >     >> >>> > > > > > is going to be going away any day now.
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > > i don't know the right way to solve this to
> >> be
> >> > > > > honest, but
> >> > > > >     >> >>> > > > > > if you
> >> > > > >     >> >>> > > have
> >> > > > >     >> >>> > > > > > ideas, i am willing to help.
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > > *Will STEVENS*
> >> > > > >     >> >>> > > > > > Lead Developer
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan
> >> > Johnson <
> >> > > > >     >> >>> njohnson@ena.com
> >> > > > >     >> >>> > >
> >> > > > >     >> >>> > > > > wrote:
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > > > > So not to re-open a can of worms, but I’m
> >> in a
> >> > > > > situation
> >> > > > >     >> >>> > > > > > > where
> >> > > > >     >> >>> I
> >> > > > >     >> >>> > > need
> >> > > > >     >> >>> > > > > > > to come up with a Marvin component test
> >> that
> >> > > > > depends on a
> >> > > > >     >> >>> > template
> >> > > > >     >> >>> > > > > > > based on a kernel that’s relatively new,
> >> i.e.,
> >> > > > > newer than
> >> > > > >     >> >>> Centos
> >> > > > >     >> >>> > > 5.3
> >> > > > >     >> >>> > > > /
> >> > > > >     >> >>> > > > > > Ubuntu 10.04 .
> >> > > > >     >> >>> > > > > > > I see openvm.eu has a suitable template
> >> > (Ubuntu
> >> > > > > 16.0.4
> >> > > > >     >> >>> > > > > > > for
> >> > > > >     >> >>> KVM),
> >> > > > >     >> >>> > > but
> >> > > > >     >> >>> > > > > > > from looking at the thread "Migrating
> >> > CloudStack
> >> > > > > content
> >> > > > >     >> >>> > > > > > > from download.cloud.com” it looks like
> >> there
> >> > is
> >> > > > >     >> >>> > > > > > > resistance to using
> >> > > > >     >> >>> > > this
> >> > > > >     >> >>> > > > at
> >> > > > >     >> >>> > > > > > > least for hosting system vm templates
> over
> >> > > > concerns
> >> > > > > of
> >> > > > >     >> >>> > neutrality.
> >> > > > >     >> >>> > > > > > > Would this be suitable for a component
> >> test?
> >> > If
> >> > > > > not,
> >> > > > >     >> >>> > > > > > > what is a
> >> > > > >     >> >>> > > > > “blessed”
> >> > > > >     >> >>> > > > > > > template location?
> >> > > > >     >> >>> > > > > > >
> >> > > > >     >> >>> > > > > > > Thanks in advance!
> >> > > > >     >> >>> > > > > > >
> >> > > > >     >> >>> > > > > > > Nathan Johnson
> >> > > > >     >> >>> > > > > > > R&D Engineer
> >> > > > >     >> >>> > > > > > >
> >> > > > >     >> >>> > > > > > >
> >> > > > >     >> >>> > > > > > >
> >> > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12
> >> Nashville,
> >> > TN
> >> > > > > 37211
> >> > > > >     >> >>> > > > > > > General Office: 615-312-6000
> >> > > > >     >> >>> > > > > > >
> >> > > > >     >> >>> > > > > > > website | blog | support
> >> > > > >     >> >>> > > > > > >
> >> > > > >     >> >>> > > > > > >
> >> > > > >     >> >>> > > > > > >
> >> > > > >     >> >>> > > > > > >
> >> > > > >     >> >>> > > > > > >
> >> > > > >     >> >>> > > > > > >
> >> > > > >     >> >>> > > > > > >
> >> > > > >     >> >>> > > > > >
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > > > DISCLAIMER
> >> > > > >     >> >>> > > > > ==========
> >> > > > >     >> >>> > > > > This e-mail may contain privileged and
> >> > confidential
> >> > > > >     >> >>> > > > > information
> >> > > > >     >> >>> which
> >> > > > >     >> >>> > > is
> >> > > > >     >> >>> > > > > the property of Accelerite, a Persistent
> >> Systems
> >> > > > > business. It
> >> > > > >     >> >>> > > > > is
> >> > > > >     >> >>> > > intended
> >> > > > >     >> >>> > > > > only for the use of the individual or entity
> to
> >> > > which
> >> > > > > it is
> >> > > > >     >> >>> > addressed.
> >> > > > >     >> >>> > > If
> >> > > > >     >> >>> > > > > you are not the intended recipient, you are
> not
> >> > > > > authorized to
> >> > > > >     >> >>> > > > > read,
> >> > > > >     >> >>> > > > retain,
> >> > > > >     >> >>> > > > > copy, print, distribute or use this message.
> If
> >> > you
> >> > > > have
> >> > > > >     >> >>> > > > > received
> >> > > > >     >> >>> > this
> >> > > > >     >> >>> > > > > communication in error, please notify the
> >> sender
> >> > and
> >> > > > > delete
> >> > > > >     >> >>> > > > > all
> >> > > > >     >> >>> > copies
> >> > > > >     >> >>> > > of
> >> > > > >     >> >>> > > > > this message. Accelerite, a Persistent
> Systems
> >> > > > business
> >> > > > > does
> >> > > > >     >> >>> > > > > not
> >> > > > >     >> >>> > accept
> >> > > > >     >> >>> > > > any
> >> > > > >     >> >>> > > > > liability for virus infected mails.
> >> > > > >     >> >>> > > > >
> >> > > > >     >> >>> > > >
> >> > > > >     >> >>> > >
> >> > > > >     >> >>> >
> >> > > > >     >> >>>
> >> > > > >     >> >
> >> > > > >     >> >abhinandan.prateek@shapeblue.com
> >> > > > >     >> >www.shapeblue.com
> >> > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >> > @shapeblue
> >> > > > >     >> >
> >> > > > >     >> >
> >> > > > >     >> >
> >> > > > >     >>
> >> > > > >     >> abhinandan.prateek@shapeblue.com
> >> > > > >     >> www.shapeblue.com
> >> > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >> > > > >     >> @shapeblue
> >> > > > >     >>
> >> > > > >     >>
> >> > > > >     >>
> >> > > > >     >>
> >> > > > >     >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > daan.hoogland@shapeblue.com
> >> > > > > www.shapeblue.com
> >> > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >> > > > > @shapeblue
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Rafael Weingärtner
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Rafael Weingärtner
> >> > >
> >> >
> >>
> >
> >
>



-- 
Rafael Weingärtner

Re: Modern template hosting

Posted by Will Stevens <sw...@apache.org>.

So now I am leaning towards the following implementation.  Use a query
parameter (more accessible than a header) to choose between delivery
options with a sane default if no query param is added.

The following (potential) options would be available:
a) [ default, no params ] current implementation.  first mirror to answer
to the web server would handle the request.
b) [ ?serve=mirrors ] return a 300 with a list of 'valid' mirrors (head
check on resource path).
c) [ ?serve=closest ] the closest 'valid' geographic mirror serves the
request.  falls back to (a) on error.
d) [ ?serve=proxy ] the web server would use (a) and stream the
corresponding mirror back to the client.

We would need a documented recommendation for users based on version
number.  The true legacy would have to use (d). It defaults to a sane
default which can handle the slow down of a mirror.  It also lays the
foundation for optimizing (d) if we need to manage the load across the
mirrors in the case of proxying.

Thoughts?  I feel like we are getting to a point where we understand what
our potential implementation paths are.

*Will Stevens*

PS - If you see this, you should consider coming to CCC in Miami.  Travel
assistance closes on March 8th!!!




On Fri, Mar 3, 2017 at 12:31 PM, Will Stevens <ws...@cloudops.com> wrote:

> 1) If the legacy implementations do not support redirects, that does cause
> a problem.  A potential solution in that case is to have the web server
> actually proxy the download, but that is not ideal and I would like to
> avoid it if possible.  Thanks for bringing that up Chiradeep.
>
> 2) I think we need to have a single URL which people can target.  Once
> they make the switch to the new URL, we want the implementation to be able
> to handle mirror failures without affecting the end client.  We want to
> avoid the situation where an ACS user will ever have to change this URL
> more than once.  Mirror failures SHOULD NOT affect the ACS users assuming
> there is still at least one mirror who can serve the requested resource.
>
> These are obviously my personal opinions and others will probably have
> differing opinions.
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal <ch...@gmail.com>
> wrote:
>
>> 1. If you are targeting legacy installations, they are not able to follow
>> redirects. The line of code that added this capability was added on 11/16.
>> 2. If you trust the users to edit the database to change the URL, you can
>> trust them to change it to anything. Just document a known list of good
>> template locations.
>>
>> On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <ws...@cloudops.com>
>> wrote:
>>
>> > So the main issue I see with this is the following.
>> >
>> > This implementation is designed to target legacy installations which
>> will
>> > be affected when download.cloud.com disappears.  These people will
>> need to
>> > make a DB change to replace the 'download.cloud.com' with some other
>> url
>> > (maybe 'download.cloudstack.org').  Once the DB has been updated, we
>> can
>> > not expect anything else of the client, they need to be able to just
>> > continue operation as they were without any need to rebuild or upgrade
>> > their ACS.
>> >
>> > If we try to force the decision to the client, then we break the legacy
>> > implementations.  Since the SSVM is likely going to be the client in
>> some
>> > cases, and since it does not already have the logic to handle the 300
>> > approach correctly, I feel like it is not a viable solution for the
>> legacy
>> > deployments.
>> >
>> > In addition to that.  I don't think the client is in any better
>> position to
>> > make the mirror decision than I am.  I am likely able to make a more
>> > 'educated' decision than the client would be able to because I can do
>> > different tests on the endpoint(s) before making a final decision.
>> >
>> > Does this make sense or am I missing something obvious?
>> >
>> > *Will STEVENS*
>> > Lead Developer
>> >
>> > <https://goo.gl/NYZ8KK>
>> >
>> > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
>> > rafaelweingartner@gmail.com> wrote:
>> >
>> > > I think I understood you, but I did not understand you.
>> > >
>> > > Let me see if I can create an example to illustrate how we could do
>> using
>> > > the HTTP 300 code.
>> > >
>> > > The HTTP 300 code indicates clients that he/she/it has multiple
>> choices
>> > to
>> > > access the requested resource. If the request is a HEAD, this is the
>> only
>> > > thing that the server returns. If the request if a GET, then the
>> server
>> > > returns the choices for the client. The response type is defined on
>> the
>> > > server based on the Content-Type, user agent and others.
>> > >
>> > > Let’s say the server receives a GET request and informs the
>> content-type
>> > as
>> > > JSON (application/JSON). Your application could get the mirrors list,
>> do
>> > > the appending process for the requested resource path, and then
>> return a
>> > > JSON array with the possible mirror the client can use. Then, the
>> client
>> > > gets this list and does the processing required to select a mirror.
>> > >
>> > > Now let’s say that a human access the link using a browser. Then, the
>> > > server should receive something like (text/plain) as the
>> content-type; I
>> > > think depending on the browser this field may differ a little (it
>> would
>> > > require some checking). Anyways, the server detects that it is a
>> “human”
>> > > requesting the resource, then we could serve an HTML page with a
>> > Javascript
>> > > that uses the mirror list. This Javascript could do some checking and
>> > > choose the best mirror for that specific.
>> > >
>> > > Why do I think this approach is interesting?
>> > > The application you developed would be used only to retrieve valid and
>> > > trusted sources of system VM images (a trusted repository for mirrors
>> > that
>> > > we as a community vouch for).  Also, we shift the decision process
>> > > regarding mirrors from the server to the client. Then, it is up to
>> > clients
>> > > to select mirrors, and not up to us (or some of our implementations).
>> > >
>> > > Did this help?
>> > > BTW: I am not saying I am against the way you proposed, which would
>> work
>> > > fine. It is merely a suggestion using a different perspective.
>> > >
>> > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
>> williamstevens@gmail.com>
>> > > wrote:
>> > >
>> > > > @rafael: in general, I don't think the client should ever be making
>> a
>> > > > choice about a mirror. We have to assume we are working with a
>> scripted
>> > > > application and anything hard coded in that implementation on the
>> > client
>> > > > side is a risk.
>> > > >
>> > > > I may not be understanding the use of the 300 approach, so I will
>> > > research
>> > > > it to see if I can make it fit.
>> > > >
>> > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
>> > > rafaelweingartner@gmail.com>
>> > > > wrote:
>> > > >
>> > > > Will great job.
>> > > >
>> > > > I had the same doubt as Daan.
>> > > >
>> > > > Have you considered using HTTP 300 working mode? Then, we could let
>> the
>> > > > client decide which mirror is the best(closest?). Section “10.3.1”
>> of
>> > [1]
>> > > > talks about it; this would put the pressure on deciding upon a
>> mirror
>> > on
>> > > > the client side, but I think it would be fair.
>> > > >
>> > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
>> > > >
>> > > > PS: You really like very short variables names!
>> > > >
>> > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
>> > > daan.hoogland@shapeblue.com
>> > > > >
>> > > > wrote:
>> > > >
>> > > > > Nice little thing Will,
>> > > > >
>> > > > > One question: if I read the code correctly it ‘go’es and tries all
>> > > > mirrors
>> > > > > at once and whichever responses first is redirected to the client.
>> > This
>> > > > > might well be the same every time. This might be the one closest
>> to
>> > the
>> > > > > mirror302 and not to the client. Is that correct or did I miss a
>> > > weighing
>> > > > > algorithm hidden in there?
>> > > > >
>> > > > > Good coding,
>> > > > >
>> > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
>> > > > >
>> > > > >     Hey All,
>> > > > >     Please review this repo and tell me what you think:
>> > > > >     https://github.com/swill/mirror302
>> > > > >
>> > > > >     Let me know if you have questions or you would like me to make
>> > > > changes
>> > > > >     before I bring this topic up with ASF Infra and open a ticket
>> to
>> > > get
>> > > > > this
>> > > > >     implemented.
>> > > > >
>> > > > >     Cheers,
>> > > > >
>> > > > >     *Will Stevens*
>> > > > >
>> > > > >
>> > > > >
>> > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
>> > > wstevens@cloudops.com
>> > > > >
>> > > > > wrote:
>> > > > >
>> > > > >     > I am building a short term solution right now.  I hopefully
>> > will
>> > > > > have the
>> > > > >     > building blocks in place by the weekend so we can start
>> working
>> > > > with
>> > > > > Infra
>> > > > >     > to get it in place.
>> > > > >     >
>> > > > >     > We will have a web server which we can point something like
>> > > > >     > downloads.cloudstack.org (or whatever url) at.  Then we
>> will
>> > > have
>> > > > > an ASF
>> > > > >     > repo which tracks a mirror list and exposes a static site
>> to be
>> > > > able
>> > > > > to
>> > > > >     > query the mirror list.
>> > > > >     >
>> > > > >     > The web server will receive a request and will do a 302
>> > redirect
>> > > to
>> > > > > the
>> > > > >     > appropriate resource on one of the mirrors.
>> > > > >     >
>> > > > >     > This gets us started.  Legacy environments will have to do
>> a DB
>> > > > > change to
>> > > > >     > change from 'download.cloud.com' to '
>> downloads.cloudstack.org'
>> > > (or
>> > > > >     > whatever), but otherwise it should be pretty simple.
>> > > > >     >
>> > > > >     > We can improve how we deliver templates going forward, but
>> this
>> > > > > obviously
>> > > > >     > requires some discussion still.  I will try to buy us some
>> time
>> > > > with
>> > > > > an
>> > > > >     > implementation which solves our problems today...
>> > > > >     >
>> > > > >     > *Will STEVENS*
>> > > > >     > Lead Developer
>> > > > >     >
>> > > > >     > <https://goo.gl/NYZ8KK>
>> > > > >     >
>> > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
>> > > > >     > abhinandan.prateek@shapeblue.com> wrote:
>> > > > >     >
>> > > > >     >>
>> > > > >     >> Initial seeding is a manual step and that is not going to
>> > change
>> > > > in
>> > > > > near
>> > > > >     >> future. A handy list of official places from where these
>> > > templates
>> > > > > can be
>> > > > >     >> downloaded won’t harm this part of the installation. Just
>> > allow
>> > > > > anyone
>> > > > >     >> credible who follows democratic process to publish their
>> > > templates
>> > > > > with
>> > > > >     >> apache’s blessings.
>> > > > >     >>
>> > > > >     >>
>> > > > >     >> Coming to urls hard coded in db, the popular suggestion is
>> to
>> > > > point
>> > > > > them
>> > > > >     >> to mirrors. This looks good but this is going to take some
>> > > effort
>> > > > > that has
>> > > > >     >> not materialised yet. Once this change is made it will be
>> > going
>> > > in
>> > > > > some
>> > > > >     >> future release like 4.11; the problem still remains for the
>> > > > > releases that
>> > > > >     >> are already out there. In case access to
>> download.cloud.com
>> > is
>> > > > > dropped
>> > > > >     >> at some point we will have to document the procedure such
>> that
>> > > > > people know
>> > > > >     >> how to make these stable releases work. That is where the
>> > > > procedure
>> > > > >     >> documented here https://shankerbalan.net/blog/
>> > > > >     >> seed-cloudstack-templates-offline/ should be officially
>> > adapted
>> > > > > with a
>> > > > >     >> marketplace to pick up templates of choosing. Maybe we can
>> > have
>> > > a
>> > > > > pre-setup
>> > > > >     >> script to make choices instead of written wiki. Note that
>> this
>> > > > will
>> > > > > be
>> > > > >     >> anyway needed by the older releases.
>> > > > >     >>
>> > > > >     >>
>> > > > >     >>
>> > > > >     >>
>> > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
>> paul.angus@shapeblue.com>
>> > > > > wrote:
>> > > > >     >>
>> > > > >     >> >-1
>> > > > >     >> >
>> > > > >     >> >We are trying to get MORE people to use CloudStack.  And
>> > > raising
>> > > > > the
>> > > > >     >> barrier to entry does nothing to help that.
>> > > > >     >> >
>> > > > >     >> >
>> > > > >     >> >Kind regards,
>> > > > >     >> >
>> > > > >     >> >Paul Angus
>> > > > >     >> >
>> > > > >     >> >paul.angus@shapeblue.com
>> > > > >     >> >www.shapeblue.com
>> > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > > > >     >> >@shapeblue
>> > > > >     >> >
>> > > > >     >> >
>> > > > >     >> >
>> > > > >     >> >
>> > > > >     >> >-----Original Message-----
>> > > > >     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@
>> > > > shapeblue.com]
>> > > > >     >> >Sent: 01 March 2017 09:39
>> > > > >     >> >To: dev@cloudstack.apache.org
>> > > > >     >> >Subject: Re: Modern template hosting
>> > > > >     >> >
>> > > > >     >> >+1
>> > > > >     >> >
>> > > > >     >> >After seeing this discussion running in circles several
>> > times,
>> > > I
>> > > > > think
>> > > > >     >> we should at least get started with the simplest option.
>> > > > >     >> >
>> > > > >     >> >
>> > > > >     >> >
>> > > > >     >> >
>> > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
>> > > chiradeepv@gmail.com>
>> > > > > wrote:
>> > > > >     >> >
>> > > > >     >> >>The alternative foolproof way is to simply not provide
>> the
>> > > > > automatic
>> > > > >     >> >>download.
>> > > > >     >> >>
>> > > > >     >> >>Just document the mirror list URL where the templates
>> can be
>> > > > > found. I
>> > > > >     >> >>don't know why people reject this option.
>> > > > >     >> >>
>> > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
>> > > > templates-offline/
>> > > > >     >> >>
>> > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
>> > > > >     >> >><wi...@gmail.com>
>> > > > >     >> >>wrote:
>> > > > >     >> >>
>> > > > >     >> >>> OK. Thanks for the heads up.
>> > > > >     >> >>>
>> > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
>> > > > > chiradeepv@gmail.com>
>> > > > >     >> wrote:
>> > > > >     >> >>>
>> > > > >     >> >>> > Sounds workable. The downloader code in the SSVM
>> won't
>> > > > follow
>> > > > >     >> >>> > redirects I think.
>> > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
>> > > > 5511065fc20787619d
>> > > > > 9cd0444
>> > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/
>> storage/template/
>> > > > >     >> >>> > HttpTemplateDownloader.java#L93
>> > > > >     >> >>> > https://goo.gl/dSi0r5
>> > > > >     >> >>> >  Might need to add
>> > > > >     >> >>> > client.setRedirectStrategy(new
>> LaxRedirectStrategy());
>> > > > >     >> >>> >
>> > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
>> > > > >     >> >>> > <ws...@cloudops.com>
>> > > > >     >> >>> > wrote:
>> > > > >     >> >>> >
>> > > > >     >> >>> > > We haven't opened a ticket yet because we don't
>> have a
>> > > > > strategy
>> > > > >     >> yet.
>> > > > >     >> >>> > >
>> > > > >     >> >>> > > What do you guys think of this:
>> > > > >     >> >>> > > - We setup a new github repo in the 'apache' org
>> which
>> > > > > consists
>> > > > >     >> >>> > > of a
>> > > > >     >> >>> > single
>> > > > >     >> >>> > > file with a list of active/supported mirrors.
>> > > > >     >> >>> > > - I write a small web server, distributed as a
>> binary,
>> > > > > which can
>> > > > >     >> >>> > > be
>> > > > >     >> >>> > hosted
>> > > > >     >> >>> > > by ASF Infra.  This web server will query the
>> current
>> > > list
>> > > > > of
>> > > > >     >> >>> > > mirrors
>> > > > >     >> >>> and
>> > > > >     >> >>> > > will select one and then do a 302 redirect to that
>> > > mirror.
>> > > > >     >> >>> > >
>> > > > >     >> >>> > > The act of 'choosing' a mirror could be done in a
>> > number
>> > > > of
>> > > > > ways.
>> > > > >     >> >>> > > - If we want to define an order, then it could just
>> > try
>> > > > > from the
>> > > > >     >> >>> > > top of
>> > > > >     >> >>> > the
>> > > > >     >> >>> > > list and work its way down.  It would curl the
>> target
>> > to
>> > > > > make
>> > > > >     >> >>> > > sure it
>> > > > >     >> >>> > gets
>> > > > >     >> >>> > > a 200 and if it does, it would do a 302 redirect.
>> > > > >     >> >>> > > - Or, if we want to distribute the load across the
>> > > > mirrors,
>> > > > > we
>> > > > >     >> >>> > > could
>> > > > >     >> >>> pick
>> > > > >     >> >>> > > from the list randomly.  Again, doing a curl to
>> verify
>> > > the
>> > > > > mirror
>> > > > >     >> >>> > > is up
>> > > > >     >> >>> > and
>> > > > >     >> >>> > > then doing a redirect.
>> > > > >     >> >>> > > - If we want to get fancy, we could do a reverse IP
>> > > lookup
>> > > > > and
>> > > > >     >> >>> > > try to
>> > > > >     >> >>> > match
>> > > > >     >> >>> > > the requester with their closest geographical
>> mirror.
>> > > > >     >> >>> > >
>> > > > >     >> >>> > > Thoughts?
>> > > > >     >> >>> > >
>> > > > >     >> >>> > > *Will STEVENS*
>> > > > >     >> >>> > > Lead Developer
>> > > > >     >> >>> > >
>> > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
>> > > > >     >> >>> > >
>> > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal
>> <
>> > > > >     >> >>> chiradeepv@gmail.com
>> > > > >     >> >>> > >
>> > > > >     >> >>> > > wrote:
>> > > > >     >> >>> > >
>> > > > >     >> >>> > > > What steps are needed to set up a mirror? What
>> does
>> > > > Infra
>> > > > > need
>> > > > >     >> to do?
>> > > > >     >> >>> > Has
>> > > > >     >> >>> > > > anybody filed a ticket with Infra?
>> > > > >     >> >>> > > >
>> > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
>> > > > >     >> >>> > > > raja.pullela@accelerite.com>
>> > > > >     >> >>> > > > wrote:
>> > > > >     >> >>> > > >
>> > > > >     >> >>> > > > > Hi will,
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > > > I believe, we didn’t get to close ‘getting a
>> > mirror
>> > > on
>> > > > > Apache’
>> > > > >     >> >>> > because
>> > > > >     >> >>> > > we
>> > > > >     >> >>> > > > > needed someone on the Apache Infra side to
>> close
>> > > this.
>> > > > > BTW,
>> > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages
>> this?)
>> > > has
>> > > > >     >> >>> > > > > all/most of
>> > > > >     >> >>> > the
>> > > > >     >> >>> > > > > content.  Once we can close on the Apache
>> mirror
>> > for
>> > > > > hosting
>> > > > >     >> >>> > > > > the
>> > > > >     >> >>> > > > content, I
>> > > > >     >> >>> > > > > can help assist getting the content there.
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > > > For now, we have replicated the
>> > download.cloud.com
>> > > > > content to
>> > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
>> > > > >     >> >>> > > > > Also, we are working on a set of
>> steps/procedure
>> > to
>> > > > > help with
>> > > > >     >> >>> > > > > this change.  I will update everyone in about a
>> > > week’s
>> > > > > time
>> > > > >     >> >>> > > > > on the
>> > > > >     >> >>> > details.
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > > > Best,
>> > > > >     >> >>> > > > > Raja Pullela
>> > > > >     >> >>> > > > > Engineering Team,
>> > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara,
>> CA,
>> > > > 95054
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "
>> williamstevens@gmail.com
>> > on
>> > > > > behalf of
>> > > > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com on
>> behalf
>> > > of
>> > > > >     >> >>> > wstevens@cloudops.com>
>> > > > >     >> >>> > > > > wrote:
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > > > unfortunately the template mirror conversation
>> got
>> > > > > caught up
>> > > > >     >> >>> > > > > in
>> > > > >     >> >>> > details
>> > > > >     >> >>> > > > and
>> > > > >     >> >>> > > > > nobody took the lead on implementing a
>> solution.
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > > > citrix has been pinging me every couple months
>> to
>> > > say
>> > > > > 'dude,
>> > > > >     >> >>> > > > > we
>> > > > >     >> >>> need
>> > > > >     >> >>> > to
>> > > > >     >> >>> > > > > remove the dependency on download.citrix.com',
>> > but
>> > > i
>> > > > > have not
>> > > > >     >> >>> > > > > had
>> > > > >     >> >>> > the
>> > > > >     >> >>> > > > > cycles to get in and solve the problem.  the
>> > > shutdown
>> > > > > of that
>> > > > >     >> >>> > > > > is
>> > > > >     >> >>> > > imminent
>> > > > >     >> >>> > > > > right now, so we need to solve it asap.
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > > > *Will STEVENS*
>> > > > >     >> >>> > > > > Lead Developer
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
>> > > > >     >> >>> > paul.angus@shapeblue.com
>> > > > >     >> >>> > > >
>> > > > >     >> >>> > > > > wrote:
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > > > > Hi Nathan,
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > > Ideally, if you put the template location in
>> (or
>> > > use
>> > > > a
>> > > > >     >> >>> > > > > > template
>> > > > >     >> >>> > > defined
>> > > > >     >> >>> > > > > > in)  test_data.py then the actual location
>> can
>> > be
>> > > > >     >> >>> > > > > > overridden by
>> > > > >     >> >>> > > anyone
>> > > > >     >> >>> > > > > > testing.
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > > For Trillian, we've copied all of the
>> templates
>> > > that
>> > > > > people
>> > > > >     >> >>> > > > > > have
>> > > > >     >> >>> > > define
>> > > > >     >> >>> > > > > to
>> > > > >     >> >>> > > > > > a local repo and then replace the URLs in
>> > > > > test_data.py to
>> > > > >     >> >>> > > > > > reduce
>> > > > >     >> >>> > > > > bandwidth
>> > > > >     >> >>> > > > > > use and download times.
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > > Ie:
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > >             "bootableIso":
>> > > > >     >> >>> > > > > >                 {
>> > > > >     >> >>> > > > > >                     "displaytext": "Test
>> > Bootable
>> > > > > ISO",
>> > > > >     >> >>> > > > > >                     "name": "testISO",
>> > > > >     >> >>> > > > > >                     "bootable": True,
>> > > > >     >> >>> > > > > >                     "ispublic": False,
>> > > > >     >> >>> > > > > >                     "url": "{{
>> > > > marvin_images_location
>> > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
>> > > > >     >> >>> > > > > >                     "ostype": 'Other Linux
>> > > > (64-bit)',
>> > > > >     >> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
>> > > > >     >> >>> > > > > >         },
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > > I thought that we had come up with a solution
>> > for
>> > > > >     >> >>> > download.cloud.com
>> > > > >     >> >>> > > ,
>> > > > >     >> >>> > > > by
>> > > > >     >> >>> > > > > > having a mirrorlist hosted in Community
>> Apache
>> > > > > 'space' with
>> > > > >     >> >>> anyone
>> > > > >     >> >>> > > able
>> > > > >     >> >>> > > > > to
>> > > > >     >> >>> > > > > > out themselves forward as a mirror.
>> > > > >     >> >>> > > > > > But I must admit I lost track of whether
>> anyone
>> > > made
>> > > > > the
>> > > > >     >> >>> requisite
>> > > > >     >> >>> > > > > changes
>> > > > >     >> >>> > > > > > in code....
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > > Kind regards,
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > > Paul Angus
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > > paul.angus@shapeblue.com
>> > > > >     >> >>> > > > > > www.shapeblue.com
>> > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N
>> > > 4HSUK
>> > > > >     >> >>> > > > > > @shapeblue
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > > -----Original Message-----
>> > > > >     >> >>> > > > > > From: williamstevens@gmail.com
>> > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
>> > > > >     >> >>> > On
>> > > > >     >> >>> > > > > > Behalf Of Will Stevens
>> > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
>> > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
>> > > > >     >> >>> > > > > > Subject: Re: Modern template hosting
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > > this is a hard questions.  in general, we
>> should
>> > > be
>> > > > > setting
>> > > > >     >> >>> > > > > > up a
>> > > > >     >> >>> > > mirror
>> > > > >     >> >>> > > > > on
>> > > > >     >> >>> > > > > > some cloudstack/apache domain and then
>> mirror to
>> > > > other
>> > > > >     >> >>> > > > > > provided
>> > > > >     >> >>> > > > > templates.
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > > we MUST come up with a solution to deprecate
>> > > > >     >> >>> > > > > > 'download.cloud.com
>> > > > >     >> >>> ',
>> > > > >     >> >>> > > > that
>> > > > >     >> >>> > > > > > is going to be going away any day now.
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > > i don't know the right way to solve this to
>> be
>> > > > > honest, but
>> > > > >     >> >>> > > > > > if you
>> > > > >     >> >>> > > have
>> > > > >     >> >>> > > > > > ideas, i am willing to help.
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > > *Will STEVENS*
>> > > > >     >> >>> > > > > > Lead Developer
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan
>> > Johnson <
>> > > > >     >> >>> njohnson@ena.com
>> > > > >     >> >>> > >
>> > > > >     >> >>> > > > > wrote:
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > > > > So not to re-open a can of worms, but I’m
>> in a
>> > > > > situation
>> > > > >     >> >>> > > > > > > where
>> > > > >     >> >>> I
>> > > > >     >> >>> > > need
>> > > > >     >> >>> > > > > > > to come up with a Marvin component test
>> that
>> > > > > depends on a
>> > > > >     >> >>> > template
>> > > > >     >> >>> > > > > > > based on a kernel that’s relatively new,
>> i.e.,
>> > > > > newer than
>> > > > >     >> >>> Centos
>> > > > >     >> >>> > > 5.3
>> > > > >     >> >>> > > > /
>> > > > >     >> >>> > > > > > Ubuntu 10.04 .
>> > > > >     >> >>> > > > > > > I see openvm.eu has a suitable template
>> > (Ubuntu
>> > > > > 16.0.4
>> > > > >     >> >>> > > > > > > for
>> > > > >     >> >>> KVM),
>> > > > >     >> >>> > > but
>> > > > >     >> >>> > > > > > > from looking at the thread "Migrating
>> > CloudStack
>> > > > > content
>> > > > >     >> >>> > > > > > > from download.cloud.com” it looks like
>> there
>> > is
>> > > > >     >> >>> > > > > > > resistance to using
>> > > > >     >> >>> > > this
>> > > > >     >> >>> > > > at
>> > > > >     >> >>> > > > > > > least for hosting system vm templates over
>> > > > concerns
>> > > > > of
>> > > > >     >> >>> > neutrality.
>> > > > >     >> >>> > > > > > > Would this be suitable for a component
>> test?
>> > If
>> > > > > not,
>> > > > >     >> >>> > > > > > > what is a
>> > > > >     >> >>> > > > > “blessed”
>> > > > >     >> >>> > > > > > > template location?
>> > > > >     >> >>> > > > > > >
>> > > > >     >> >>> > > > > > > Thanks in advance!
>> > > > >     >> >>> > > > > > >
>> > > > >     >> >>> > > > > > > Nathan Johnson
>> > > > >     >> >>> > > > > > > R&D Engineer
>> > > > >     >> >>> > > > > > >
>> > > > >     >> >>> > > > > > >
>> > > > >     >> >>> > > > > > >
>> > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12
>> Nashville,
>> > TN
>> > > > > 37211
>> > > > >     >> >>> > > > > > > General Office: 615-312-6000
>> > > > >     >> >>> > > > > > >
>> > > > >     >> >>> > > > > > > website | blog | support
>> > > > >     >> >>> > > > > > >
>> > > > >     >> >>> > > > > > >
>> > > > >     >> >>> > > > > > >
>> > > > >     >> >>> > > > > > >
>> > > > >     >> >>> > > > > > >
>> > > > >     >> >>> > > > > > >
>> > > > >     >> >>> > > > > > >
>> > > > >     >> >>> > > > > >
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > > > DISCLAIMER
>> > > > >     >> >>> > > > > ==========
>> > > > >     >> >>> > > > > This e-mail may contain privileged and
>> > confidential
>> > > > >     >> >>> > > > > information
>> > > > >     >> >>> which
>> > > > >     >> >>> > > is
>> > > > >     >> >>> > > > > the property of Accelerite, a Persistent
>> Systems
>> > > > > business. It
>> > > > >     >> >>> > > > > is
>> > > > >     >> >>> > > intended
>> > > > >     >> >>> > > > > only for the use of the individual or entity to
>> > > which
>> > > > > it is
>> > > > >     >> >>> > addressed.
>> > > > >     >> >>> > > If
>> > > > >     >> >>> > > > > you are not the intended recipient, you are not
>> > > > > authorized to
>> > > > >     >> >>> > > > > read,
>> > > > >     >> >>> > > > retain,
>> > > > >     >> >>> > > > > copy, print, distribute or use this message. If
>> > you
>> > > > have
>> > > > >     >> >>> > > > > received
>> > > > >     >> >>> > this
>> > > > >     >> >>> > > > > communication in error, please notify the
>> sender
>> > and
>> > > > > delete
>> > > > >     >> >>> > > > > all
>> > > > >     >> >>> > copies
>> > > > >     >> >>> > > of
>> > > > >     >> >>> > > > > this message. Accelerite, a Persistent Systems
>> > > > business
>> > > > > does
>> > > > >     >> >>> > > > > not
>> > > > >     >> >>> > accept
>> > > > >     >> >>> > > > any
>> > > > >     >> >>> > > > > liability for virus infected mails.
>> > > > >     >> >>> > > > >
>> > > > >     >> >>> > > >
>> > > > >     >> >>> > >
>> > > > >     >> >>> >
>> > > > >     >> >>>
>> > > > >     >> >
>> > > > >     >> >abhinandan.prateek@shapeblue.com
>> > > > >     >> >www.shapeblue.com
>> > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > @shapeblue
>> > > > >     >> >
>> > > > >     >> >
>> > > > >     >> >
>> > > > >     >>
>> > > > >     >> abhinandan.prateek@shapeblue.com
>> > > > >     >> www.shapeblue.com
>> > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > > > >     >> @shapeblue
>> > > > >     >>
>> > > > >     >>
>> > > > >     >>
>> > > > >     >>
>> > > > >     >
>> > > > >
>> > > > >
>> > > > >
>> > > > > daan.hoogland@shapeblue.com
>> > > > > www.shapeblue.com
>> > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > > > > @shapeblue
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > Rafael Weingärtner
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Rafael Weingärtner
>> > >
>> >
>>
>
>

Re: Modern template hosting

Posted by Will Stevens <ws...@cloudops.com>.

yes, system vm templates only.  solve for legacy setups when
download.cloud.com goes away.  optionally, give the community more control
over available mirrors without impacting the documentation and setup
process.

a valid option i think we should still be considering is just trying to get
the ASF to point 'downloads.cloudstack.org' at wido's repository.  at least
we then have basic 'control' over the domain which we are using in the
documentation.

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Fri, Mar 3, 2017 at 1:43 PM, Rafael Weingärtner <
rafaelweingartner@gmail.com> wrote:

> I think so far the mirror scheme we are discussing is not for general
> purpose VMs, right? I was under the impression that we wanted to improve
> the system VMs template management scheme ONLY.
>
> On Fri, Mar 3, 2017 at 1:27 PM, Chiradeep Vittal <ch...@gmail.com>
> wrote:
>
> > I do feel like this is early optimization. Mirrors rarely fail. I'd
> expect
> > a single web server hosted on Apache Infra without any monitors to fail
> > more often than a mirror. We already expect Wido's systemvm repository to
> > be up all the time. And it has been. Similarly, I don't believe Nux's
> > repository has ever been down. And if Accelerite wants to host on S3,
> that
> > one is pretty solid as well.
> >
> > This is an infrequent operation in a cloud. After the cloud is installed,
> > the download servers are only needed for a new zone. If we trust the user
> > to run a cloud, surely he/she can run a web server to serve some built-in
> > templates. And if her cloud is successful, she needs to figure out how to
> > host her templates anyway and not rely on 3rd parties.
> >
> >
> > On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <ws...@cloudops.com>
> > wrote:
> >
> > > 1) If the legacy implementations do not support redirects, that does
> > cause
> > > a problem.  A potential solution in that case is to have the web server
> > > actually proxy the download, but that is not ideal and I would like to
> > > avoid it if possible.  Thanks for bringing that up Chiradeep.
> > >
> > > 2) I think we need to have a single URL which people can target.  Once
> > they
> > > make the switch to the new URL, we want the implementation to be able
> to
> > > handle mirror failures without affecting the end client.  We want to
> > avoid
> > > the situation where an ACS user will ever have to change this URL more
> > than
> > > once.  Mirror failures SHOULD NOT affect the ACS users assuming there
> is
> > > still at least one mirror who can serve the requested resource.
> > >
> > > These are obviously my personal opinions and others will probably have
> > > differing opinions.
> > >
> > > *Will STEVENS*
> > > Lead Developer
> > >
> > > <https://goo.gl/NYZ8KK>
> > >
> > > On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal <
> chiradeepv@gmail.com>
> > > wrote:
> > >
> > > > 1. If you are targeting legacy installations, they are not able to
> > follow
> > > > redirects. The line of code that added this capability was added on
> > > 11/16.
> > > > 2. If you trust the users to edit the database to change the URL, you
> > can
> > > > trust them to change it to anything. Just document a known list of
> good
> > > > template locations.
> > > >
> > > > On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <ws...@cloudops.com>
> > > > wrote:
> > > >
> > > > > So the main issue I see with this is the following.
> > > > >
> > > > > This implementation is designed to target legacy installations
> which
> > > will
> > > > > be affected when download.cloud.com disappears.  These people will
> > > need
> > > > to
> > > > > make a DB change to replace the 'download.cloud.com' with some
> other
> > > url
> > > > > (maybe 'download.cloudstack.org').  Once the DB has been updated,
> we
> > > can
> > > > > not expect anything else of the client, they need to be able to
> just
> > > > > continue operation as they were without any need to rebuild or
> > upgrade
> > > > > their ACS.
> > > > >
> > > > > If we try to force the decision to the client, then we break the
> > legacy
> > > > > implementations.  Since the SSVM is likely going to be the client
> in
> > > some
> > > > > cases, and since it does not already have the logic to handle the
> 300
> > > > > approach correctly, I feel like it is not a viable solution for the
> > > > legacy
> > > > > deployments.
> > > > >
> > > > > In addition to that.  I don't think the client is in any better
> > > position
> > > > to
> > > > > make the mirror decision than I am.  I am likely able to make a
> more
> > > > > 'educated' decision than the client would be able to because I can
> do
> > > > > different tests on the endpoint(s) before making a final decision.
> > > > >
> > > > > Does this make sense or am I missing something obvious?
> > > > >
> > > > > *Will STEVENS*
> > > > > Lead Developer
> > > > >
> > > > > <https://goo.gl/NYZ8KK>
> > > > >
> > > > > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
> > > > > rafaelweingartner@gmail.com> wrote:
> > > > >
> > > > > > I think I understood you, but I did not understand you.
> > > > > >
> > > > > > Let me see if I can create an example to illustrate how we could
> do
> > > > using
> > > > > > the HTTP 300 code.
> > > > > >
> > > > > > The HTTP 300 code indicates clients that he/she/it has multiple
> > > choices
> > > > > to
> > > > > > access the requested resource. If the request is a HEAD, this is
> > the
> > > > only
> > > > > > thing that the server returns. If the request if a GET, then the
> > > server
> > > > > > returns the choices for the client. The response type is defined
> on
> > > the
> > > > > > server based on the Content-Type, user agent and others.
> > > > > >
> > > > > > Let’s say the server receives a GET request and informs the
> > > > content-type
> > > > > as
> > > > > > JSON (application/JSON). Your application could get the mirrors
> > list,
> > > > do
> > > > > > the appending process for the requested resource path, and then
> > > return
> > > > a
> > > > > > JSON array with the possible mirror the client can use. Then, the
> > > > client
> > > > > > gets this list and does the processing required to select a
> mirror.
> > > > > >
> > > > > > Now let’s say that a human access the link using a browser. Then,
> > the
> > > > > > server should receive something like (text/plain) as the
> > > content-type;
> > > > I
> > > > > > think depending on the browser this field may differ a little (it
> > > would
> > > > > > require some checking). Anyways, the server detects that it is a
> > > > “human”
> > > > > > requesting the resource, then we could serve an HTML page with a
> > > > > Javascript
> > > > > > that uses the mirror list. This Javascript could do some checking
> > and
> > > > > > choose the best mirror for that specific.
> > > > > >
> > > > > > Why do I think this approach is interesting?
> > > > > > The application you developed would be used only to retrieve
> valid
> > > and
> > > > > > trusted sources of system VM images (a trusted repository for
> > mirrors
> > > > > that
> > > > > > we as a community vouch for).  Also, we shift the decision
> process
> > > > > > regarding mirrors from the server to the client. Then, it is up
> to
> > > > > clients
> > > > > > to select mirrors, and not up to us (or some of our
> > implementations).
> > > > > >
> > > > > > Did this help?
> > > > > > BTW: I am not saying I am against the way you proposed, which
> would
> > > > work
> > > > > > fine. It is merely a suggestion using a different perspective.
> > > > > >
> > > > > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
> > > > williamstevens@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > @rafael: in general, I don't think the client should ever be
> > > making a
> > > > > > > choice about a mirror. We have to assume we are working with a
> > > > scripted
> > > > > > > application and anything hard coded in that implementation on
> the
> > > > > client
> > > > > > > side is a risk.
> > > > > > >
> > > > > > > I may not be understanding the use of the 300 approach, so I
> will
> > > > > > research
> > > > > > > it to see if I can make it fit.
> > > > > > >
> > > > > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
> > > > > > rafaelweingartner@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > Will great job.
> > > > > > >
> > > > > > > I had the same doubt as Daan.
> > > > > > >
> > > > > > > Have you considered using HTTP 300 working mode? Then, we could
> > let
> > > > the
> > > > > > > client decide which mirror is the best(closest?). Section
> > “10.3.1”
> > > of
> > > > > [1]
> > > > > > > talks about it; this would put the pressure on deciding upon a
> > > mirror
> > > > > on
> > > > > > > the client side, but I think it would be fair.
> > > > > > >
> > > > > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
> > > > > > >
> > > > > > > PS: You really like very short variables names!
> > > > > > >
> > > > > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
> > > > > > daan.hoogland@shapeblue.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Nice little thing Will,
> > > > > > > >
> > > > > > > > One question: if I read the code correctly it ‘go’es and
> tries
> > > all
> > > > > > > mirrors
> > > > > > > > at once and whichever responses first is redirected to the
> > > client.
> > > > > This
> > > > > > > > might well be the same every time. This might be the one
> > closest
> > > to
> > > > > the
> > > > > > > > mirror302 and not to the client. Is that correct or did I
> miss
> > a
> > > > > > weighing
> > > > > > > > algorithm hidden in there?
> > > > > > > >
> > > > > > > > Good coding,
> > > > > > > >
> > > > > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
> > > > > > > >
> > > > > > > >     Hey All,
> > > > > > > >     Please review this repo and tell me what you think:
> > > > > > > >     https://github.com/swill/mirror302
> > > > > > > >
> > > > > > > >     Let me know if you have questions or you would like me to
> > > make
> > > > > > > changes
> > > > > > > >     before I bring this topic up with ASF Infra and open a
> > ticket
> > > > to
> > > > > > get
> > > > > > > > this
> > > > > > > >     implemented.
> > > > > > > >
> > > > > > > >     Cheers,
> > > > > > > >
> > > > > > > >     *Will Stevens*
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> > > > > > wstevens@cloudops.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >     > I am building a short term solution right now.  I
> > hopefully
> > > > > will
> > > > > > > > have the
> > > > > > > >     > building blocks in place by the weekend so we can start
> > > > working
> > > > > > > with
> > > > > > > > Infra
> > > > > > > >     > to get it in place.
> > > > > > > >     >
> > > > > > > >     > We will have a web server which we can point something
> > like
> > > > > > > >     > downloads.cloudstack.org (or whatever url) at.  Then
> we
> > > will
> > > > > > have
> > > > > > > > an ASF
> > > > > > > >     > repo which tracks a mirror list and exposes a static
> site
> > > to
> > > > be
> > > > > > > able
> > > > > > > > to
> > > > > > > >     > query the mirror list.
> > > > > > > >     >
> > > > > > > >     > The web server will receive a request and will do a 302
> > > > > redirect
> > > > > > to
> > > > > > > > the
> > > > > > > >     > appropriate resource on one of the mirrors.
> > > > > > > >     >
> > > > > > > >     > This gets us started.  Legacy environments will have to
> > do
> > > a
> > > > DB
> > > > > > > > change to
> > > > > > > >     > change from 'download.cloud.com' to '
> > > > downloads.cloudstack.org'
> > > > > > (or
> > > > > > > >     > whatever), but otherwise it should be pretty simple.
> > > > > > > >     >
> > > > > > > >     > We can improve how we deliver templates going forward,
> > but
> > > > this
> > > > > > > > obviously
> > > > > > > >     > requires some discussion still.  I will try to buy us
> > some
> > > > time
> > > > > > > with
> > > > > > > > an
> > > > > > > >     > implementation which solves our problems today...
> > > > > > > >     >
> > > > > > > >     > *Will STEVENS*
> > > > > > > >     > Lead Developer
> > > > > > > >     >
> > > > > > > >     > <https://goo.gl/NYZ8KK>
> > > > > > > >     >
> > > > > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > > > > > > >     > abhinandan.prateek@shapeblue.com> wrote:
> > > > > > > >     >
> > > > > > > >     >>
> > > > > > > >     >> Initial seeding is a manual step and that is not going
> > to
> > > > > change
> > > > > > > in
> > > > > > > > near
> > > > > > > >     >> future. A handy list of official places from where
> these
> > > > > > templates
> > > > > > > > can be
> > > > > > > >     >> downloaded won’t harm this part of the installation.
> > Just
> > > > > allow
> > > > > > > > anyone
> > > > > > > >     >> credible who follows democratic process to publish
> their
> > > > > > templates
> > > > > > > > with
> > > > > > > >     >> apache’s blessings.
> > > > > > > >     >>
> > > > > > > >     >>
> > > > > > > >     >> Coming to urls hard coded in db, the popular
> suggestion
> > is
> > > > to
> > > > > > > point
> > > > > > > > them
> > > > > > > >     >> to mirrors. This looks good but this is going to take
> > some
> > > > > > effort
> > > > > > > > that has
> > > > > > > >     >> not materialised yet. Once this change is made it will
> > be
> > > > > going
> > > > > > in
> > > > > > > > some
> > > > > > > >     >> future release like 4.11; the problem still remains
> for
> > > the
> > > > > > > > releases that
> > > > > > > >     >> are already out there. In case access to
> > > download.cloud.com
> > > > > is
> > > > > > > > dropped
> > > > > > > >     >> at some point we will have to document the procedure
> > such
> > > > that
> > > > > > > > people know
> > > > > > > >     >> how to make these stable releases work. That is where
> > the
> > > > > > > procedure
> > > > > > > >     >> documented here https://shankerbalan.net/blog/
> > > > > > > >     >> seed-cloudstack-templates-offline/ should be
> officially
> > > > > adapted
> > > > > > > > with a
> > > > > > > >     >> marketplace to pick up templates of choosing. Maybe we
> > can
> > > > > have
> > > > > > a
> > > > > > > > pre-setup
> > > > > > > >     >> script to make choices instead of written wiki. Note
> > that
> > > > this
> > > > > > > will
> > > > > > > > be
> > > > > > > >     >> anyway needed by the older releases.
> > > > > > > >     >>
> > > > > > > >     >>
> > > > > > > >     >>
> > > > > > > >     >>
> > > > > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
> > > > paul.angus@shapeblue.com>
> > > > > > > > wrote:
> > > > > > > >     >>
> > > > > > > >     >> >-1
> > > > > > > >     >> >
> > > > > > > >     >> >We are trying to get MORE people to use CloudStack.
> > And
> > > > > > raising
> > > > > > > > the
> > > > > > > >     >> barrier to entry does nothing to help that.
> > > > > > > >     >> >
> > > > > > > >     >> >
> > > > > > > >     >> >Kind regards,
> > > > > > > >     >> >
> > > > > > > >     >> >Paul Angus
> > > > > > > >     >> >
> > > > > > > >     >> >paul.angus@shapeblue.com
> > > > > > > >     >> >www.shapeblue.com
> > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > >     >> >@shapeblue
> > > > > > > >     >> >
> > > > > > > >     >> >
> > > > > > > >     >> >
> > > > > > > >     >> >
> > > > > > > >     >> >-----Original Message-----
> > > > > > > >     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@
> > > > > > > shapeblue.com]
> > > > > > > >     >> >Sent: 01 March 2017 09:39
> > > > > > > >     >> >To: dev@cloudstack.apache.org
> > > > > > > >     >> >Subject: Re: Modern template hosting
> > > > > > > >     >> >
> > > > > > > >     >> >+1
> > > > > > > >     >> >
> > > > > > > >     >> >After seeing this discussion running in circles
> several
> > > > > times,
> > > > > > I
> > > > > > > > think
> > > > > > > >     >> we should at least get started with the simplest
> option.
> > > > > > > >     >> >
> > > > > > > >     >> >
> > > > > > > >     >> >
> > > > > > > >     >> >
> > > > > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
> > > > > > chiradeepv@gmail.com>
> > > > > > > > wrote:
> > > > > > > >     >> >
> > > > > > > >     >> >>The alternative foolproof way is to simply not
> provide
> > > the
> > > > > > > > automatic
> > > > > > > >     >> >>download.
> > > > > > > >     >> >>
> > > > > > > >     >> >>Just document the mirror list URL where the
> templates
> > > can
> > > > be
> > > > > > > > found. I
> > > > > > > >     >> >>don't know why people reject this option.
> > > > > > > >     >> >>
> > > > > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
> > > > > > > templates-offline/
> > > > > > > >     >> >>
> > > > > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> > > > > > > >     >> >><wi...@gmail.com>
> > > > > > > >     >> >>wrote:
> > > > > > > >     >> >>
> > > > > > > >     >> >>> OK. Thanks for the heads up.
> > > > > > > >     >> >>>
> > > > > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > > > > > chiradeepv@gmail.com>
> > > > > > > >     >> wrote:
> > > > > > > >     >> >>>
> > > > > > > >     >> >>> > Sounds workable. The downloader code in the SSVM
> > > won't
> > > > > > > follow
> > > > > > > >     >> >>> > redirects I think.
> > > > > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
> > > > > > > 5511065fc20787619d
> > > > > > > > 9cd0444
> > > > > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/
> > > storage/template/
> > > > > > > >     >> >>> > HttpTemplateDownloader.java#L93
> > > > > > > >     >> >>> > https://goo.gl/dSi0r5
> > > > > > > >     >> >>> >  Might need to add
> > > > > > > >     >> >>> > client.setRedirectStrategy(new
> > > LaxRedirectStrategy());
> > > > > > > >     >> >>> >
> > > > > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> > > > > > > >     >> >>> > <ws...@cloudops.com>
> > > > > > > >     >> >>> > wrote:
> > > > > > > >     >> >>> >
> > > > > > > >     >> >>> > > We haven't opened a ticket yet because we
> don't
> > > > have a
> > > > > > > > strategy
> > > > > > > >     >> yet.
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> > > What do you guys think of this:
> > > > > > > >     >> >>> > > - We setup a new github repo in the 'apache'
> org
> > > > which
> > > > > > > > consists
> > > > > > > >     >> >>> > > of a
> > > > > > > >     >> >>> > single
> > > > > > > >     >> >>> > > file with a list of active/supported mirrors.
> > > > > > > >     >> >>> > > - I write a small web server, distributed as a
> > > > binary,
> > > > > > > > which can
> > > > > > > >     >> >>> > > be
> > > > > > > >     >> >>> > hosted
> > > > > > > >     >> >>> > > by ASF Infra.  This web server will query the
> > > > current
> > > > > > list
> > > > > > > > of
> > > > > > > >     >> >>> > > mirrors
> > > > > > > >     >> >>> and
> > > > > > > >     >> >>> > > will select one and then do a 302 redirect to
> > that
> > > > > > mirror.
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> > > The act of 'choosing' a mirror could be done
> in
> > a
> > > > > number
> > > > > > > of
> > > > > > > > ways.
> > > > > > > >     >> >>> > > - If we want to define an order, then it could
> > > just
> > > > > try
> > > > > > > > from the
> > > > > > > >     >> >>> > > top of
> > > > > > > >     >> >>> > the
> > > > > > > >     >> >>> > > list and work its way down.  It would curl the
> > > > target
> > > > > to
> > > > > > > > make
> > > > > > > >     >> >>> > > sure it
> > > > > > > >     >> >>> > gets
> > > > > > > >     >> >>> > > a 200 and if it does, it would do a 302
> > redirect.
> > > > > > > >     >> >>> > > - Or, if we want to distribute the load across
> > the
> > > > > > > mirrors,
> > > > > > > > we
> > > > > > > >     >> >>> > > could
> > > > > > > >     >> >>> pick
> > > > > > > >     >> >>> > > from the list randomly.  Again, doing a curl
> to
> > > > verify
> > > > > > the
> > > > > > > > mirror
> > > > > > > >     >> >>> > > is up
> > > > > > > >     >> >>> > and
> > > > > > > >     >> >>> > > then doing a redirect.
> > > > > > > >     >> >>> > > - If we want to get fancy, we could do a
> reverse
> > > IP
> > > > > > lookup
> > > > > > > > and
> > > > > > > >     >> >>> > > try to
> > > > > > > >     >> >>> > match
> > > > > > > >     >> >>> > > the requester with their closest geographical
> > > > mirror.
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> > > Thoughts?
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> > > *Will STEVENS*
> > > > > > > >     >> >>> > > Lead Developer
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep
> > > Vittal <
> > > > > > > >     >> >>> chiradeepv@gmail.com
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> > > wrote:
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> > > > What steps are needed to set up a mirror?
> What
> > > > does
> > > > > > > Infra
> > > > > > > > need
> > > > > > > >     >> to do?
> > > > > > > >     >> >>> > Has
> > > > > > > >     >> >>> > > > anybody filed a ticket with Infra?
> > > > > > > >     >> >>> > > >
> > > > > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja
> > Pullela <
> > > > > > > >     >> >>> > > > raja.pullela@accelerite.com>
> > > > > > > >     >> >>> > > > wrote:
> > > > > > > >     >> >>> > > >
> > > > > > > >     >> >>> > > > > Hi will,
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > I believe, we didn’t get to close
> ‘getting a
> > > > > mirror
> > > > > > on
> > > > > > > > Apache’
> > > > > > > >     >> >>> > because
> > > > > > > >     >> >>> > > we
> > > > > > > >     >> >>> > > > > needed someone on the Apache Infra side to
> > > close
> > > > > > this.
> > > > > > > > BTW,
> > > > > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux
> manages
> > > > this?)
> > > > > > has
> > > > > > > >     >> >>> > > > > all/most of
> > > > > > > >     >> >>> > the
> > > > > > > >     >> >>> > > > > content.  Once we can close on the Apache
> > > mirror
> > > > > for
> > > > > > > > hosting
> > > > > > > >     >> >>> > > > > the
> > > > > > > >     >> >>> > > > content, I
> > > > > > > >     >> >>> > > > > can help assist getting the content there.
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > For now, we have replicated the
> > > > > download.cloud.com
> > > > > > > > content to
> > > > > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
> > > > > > > >     >> >>> > > > > Also, we are working on a set of
> > > steps/procedure
> > > > > to
> > > > > > > > help with
> > > > > > > >     >> >>> > > > > this change.  I will update everyone in
> > about
> > > a
> > > > > > week’s
> > > > > > > > time
> > > > > > > >     >> >>> > > > > on the
> > > > > > > >     >> >>> > details.
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > Best,
> > > > > > > >     >> >>> > > > > Raja Pullela
> > > > > > > >     >> >>> > > > > Engineering Team,
> > > > > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa
> > Clara,
> > > > CA,
> > > > > > > 95054
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "
> > > williamstevens@gmail.com
> > > > > on
> > > > > > > > behalf of
> > > > > > > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com
> on
> > > > behalf
> > > > > > of
> > > > > > > >     >> >>> > wstevens@cloudops.com>
> > > > > > > >     >> >>> > > > > wrote:
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > unfortunately the template mirror
> > conversation
> > > > got
> > > > > > > > caught up
> > > > > > > >     >> >>> > > > > in
> > > > > > > >     >> >>> > details
> > > > > > > >     >> >>> > > > and
> > > > > > > >     >> >>> > > > > nobody took the lead on implementing a
> > > solution.
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > citrix has been pinging me every couple
> > months
> > > > to
> > > > > > say
> > > > > > > > 'dude,
> > > > > > > >     >> >>> > > > > we
> > > > > > > >     >> >>> need
> > > > > > > >     >> >>> > to
> > > > > > > >     >> >>> > > > > remove the dependency on
> > download.citrix.com
> > > ',
> > > > > but
> > > > > > i
> > > > > > > > have not
> > > > > > > >     >> >>> > > > > had
> > > > > > > >     >> >>> > the
> > > > > > > >     >> >>> > > > > cycles to get in and solve the problem.
> the
> > > > > > shutdown
> > > > > > > > of that
> > > > > > > >     >> >>> > > > > is
> > > > > > > >     >> >>> > > imminent
> > > > > > > >     >> >>> > > > > right now, so we need to solve it asap.
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > *Will STEVENS*
> > > > > > > >     >> >>> > > > > Lead Developer
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul
> > Angus <
> > > > > > > >     >> >>> > paul.angus@shapeblue.com
> > > > > > > >     >> >>> > > >
> > > > > > > >     >> >>> > > > > wrote:
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > > Hi Nathan,
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > Ideally, if you put the template
> location
> > in
> > > > (or
> > > > > > use
> > > > > > > a
> > > > > > > >     >> >>> > > > > > template
> > > > > > > >     >> >>> > > defined
> > > > > > > >     >> >>> > > > > > in)  test_data.py then the actual
> location
> > > can
> > > > > be
> > > > > > > >     >> >>> > > > > > overridden by
> > > > > > > >     >> >>> > > anyone
> > > > > > > >     >> >>> > > > > > testing.
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > For Trillian, we've copied all of the
> > > > templates
> > > > > > that
> > > > > > > > people
> > > > > > > >     >> >>> > > > > > have
> > > > > > > >     >> >>> > > define
> > > > > > > >     >> >>> > > > > to
> > > > > > > >     >> >>> > > > > > a local repo and then replace the URLs
> in
> > > > > > > > test_data.py to
> > > > > > > >     >> >>> > > > > > reduce
> > > > > > > >     >> >>> > > > > bandwidth
> > > > > > > >     >> >>> > > > > > use and download times.
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > Ie:
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >             "bootableIso":
> > > > > > > >     >> >>> > > > > >                 {
> > > > > > > >     >> >>> > > > > >                     "displaytext": "Test
> > > > > Bootable
> > > > > > > > ISO",
> > > > > > > >     >> >>> > > > > >                     "name": "testISO",
> > > > > > > >     >> >>> > > > > >                     "bootable": True,
> > > > > > > >     >> >>> > > > > >                     "ispublic": False,
> > > > > > > >     >> >>> > > > > >                     "url": "{{
> > > > > > > marvin_images_location
> > > > > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
> > > > > > > >     >> >>> > > > > >                     "ostype": 'Other
> Linux
> > > > > > > (64-bit)',
> > > > > > > >     >> >>> > > > > >                     "mode":
> > 'HTTP_DOWNLOAD'
> > > > > > > >     >> >>> > > > > >         },
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > I thought that we had come up with a
> > > solution
> > > > > for
> > > > > > > >     >> >>> > download.cloud.com
> > > > > > > >     >> >>> > > ,
> > > > > > > >     >> >>> > > > by
> > > > > > > >     >> >>> > > > > > having a mirrorlist hosted in Community
> > > Apache
> > > > > > > > 'space' with
> > > > > > > >     >> >>> anyone
> > > > > > > >     >> >>> > > able
> > > > > > > >     >> >>> > > > > to
> > > > > > > >     >> >>> > > > > > out themselves forward as a mirror.
> > > > > > > >     >> >>> > > > > > But I must admit I lost track of whether
> > > > anyone
> > > > > > made
> > > > > > > > the
> > > > > > > >     >> >>> requisite
> > > > > > > >     >> >>> > > > > changes
> > > > > > > >     >> >>> > > > > > in code....
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > Kind regards,
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > Paul Angus
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > paul.angus@shapeblue.com
> > > > > > > >     >> >>> > > > > > www.shapeblue.com
> > > > > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden, London
> > > WC2N
> > > > > > 4HSUK
> > > > > > > >     >> >>> > > > > > @shapeblue
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > -----Original Message-----
> > > > > > > >     >> >>> > > > > > From: williamstevens@gmail.com
> > > > > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
> > > > > > > >     >> >>> > On
> > > > > > > >     >> >>> > > > > > Behalf Of Will Stevens
> > > > > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
> > > > > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
> > > > > > > >     >> >>> > > > > > Subject: Re: Modern template hosting
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > this is a hard questions.  in general,
> we
> > > > should
> > > > > > be
> > > > > > > > setting
> > > > > > > >     >> >>> > > > > > up a
> > > > > > > >     >> >>> > > mirror
> > > > > > > >     >> >>> > > > > on
> > > > > > > >     >> >>> > > > > > some cloudstack/apache domain and then
> > > mirror
> > > > to
> > > > > > > other
> > > > > > > >     >> >>> > > > > > provided
> > > > > > > >     >> >>> > > > > templates.
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > we MUST come up with a solution to
> > deprecate
> > > > > > > >     >> >>> > > > > > 'download.cloud.com
> > > > > > > >     >> >>> ',
> > > > > > > >     >> >>> > > > that
> > > > > > > >     >> >>> > > > > > is going to be going away any day now.
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > i don't know the right way to solve this
> > to
> > > be
> > > > > > > > honest, but
> > > > > > > >     >> >>> > > > > > if you
> > > > > > > >     >> >>> > > have
> > > > > > > >     >> >>> > > > > > ideas, i am willing to help.
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > *Will STEVENS*
> > > > > > > >     >> >>> > > > > > Lead Developer
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan
> > > > > Johnson <
> > > > > > > >     >> >>> njohnson@ena.com
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> > > > > wrote:
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > > So not to re-open a can of worms, but
> > I’m
> > > > in a
> > > > > > > > situation
> > > > > > > >     >> >>> > > > > > > where
> > > > > > > >     >> >>> I
> > > > > > > >     >> >>> > > need
> > > > > > > >     >> >>> > > > > > > to come up with a Marvin component
> test
> > > that
> > > > > > > > depends on a
> > > > > > > >     >> >>> > template
> > > > > > > >     >> >>> > > > > > > based on a kernel that’s relatively
> new,
> > > > i.e.,
> > > > > > > > newer than
> > > > > > > >     >> >>> Centos
> > > > > > > >     >> >>> > > 5.3
> > > > > > > >     >> >>> > > > /
> > > > > > > >     >> >>> > > > > > Ubuntu 10.04 .
> > > > > > > >     >> >>> > > > > > > I see openvm.eu has a suitable
> template
> > > > > (Ubuntu
> > > > > > > > 16.0.4
> > > > > > > >     >> >>> > > > > > > for
> > > > > > > >     >> >>> KVM),
> > > > > > > >     >> >>> > > but
> > > > > > > >     >> >>> > > > > > > from looking at the thread "Migrating
> > > > > CloudStack
> > > > > > > > content
> > > > > > > >     >> >>> > > > > > > from download.cloud.com” it looks
> like
> > > > there
> > > > > is
> > > > > > > >     >> >>> > > > > > > resistance to using
> > > > > > > >     >> >>> > > this
> > > > > > > >     >> >>> > > > at
> > > > > > > >     >> >>> > > > > > > least for hosting system vm templates
> > over
> > > > > > > concerns
> > > > > > > > of
> > > > > > > >     >> >>> > neutrality.
> > > > > > > >     >> >>> > > > > > > Would this be suitable for a component
> > > test?
> > > > > If
> > > > > > > > not,
> > > > > > > >     >> >>> > > > > > > what is a
> > > > > > > >     >> >>> > > > > “blessed”
> > > > > > > >     >> >>> > > > > > > template location?
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > > Thanks in advance!
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > > Nathan Johnson
> > > > > > > >     >> >>> > > > > > > R&D Engineer
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > Nashville,
> > > > > TN
> > > > > > > > 37211
> > > > > > > >     >> >>> > > > > > > General Office: 615-312-6000
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > > website | blog | support
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > DISCLAIMER
> > > > > > > >     >> >>> > > > > ==========
> > > > > > > >     >> >>> > > > > This e-mail may contain privileged and
> > > > > confidential
> > > > > > > >     >> >>> > > > > information
> > > > > > > >     >> >>> which
> > > > > > > >     >> >>> > > is
> > > > > > > >     >> >>> > > > > the property of Accelerite, a Persistent
> > > Systems
> > > > > > > > business. It
> > > > > > > >     >> >>> > > > > is
> > > > > > > >     >> >>> > > intended
> > > > > > > >     >> >>> > > > > only for the use of the individual or
> entity
> > > to
> > > > > > which
> > > > > > > > it is
> > > > > > > >     >> >>> > addressed.
> > > > > > > >     >> >>> > > If
> > > > > > > >     >> >>> > > > > you are not the intended recipient, you
> are
> > > not
> > > > > > > > authorized to
> > > > > > > >     >> >>> > > > > read,
> > > > > > > >     >> >>> > > > retain,
> > > > > > > >     >> >>> > > > > copy, print, distribute or use this
> message.
> > > If
> > > > > you
> > > > > > > have
> > > > > > > >     >> >>> > > > > received
> > > > > > > >     >> >>> > this
> > > > > > > >     >> >>> > > > > communication in error, please notify the
> > > sender
> > > > > and
> > > > > > > > delete
> > > > > > > >     >> >>> > > > > all
> > > > > > > >     >> >>> > copies
> > > > > > > >     >> >>> > > of
> > > > > > > >     >> >>> > > > > this message. Accelerite, a Persistent
> > Systems
> > > > > > > business
> > > > > > > > does
> > > > > > > >     >> >>> > > > > not
> > > > > > > >     >> >>> > accept
> > > > > > > >     >> >>> > > > any
> > > > > > > >     >> >>> > > > > liability for virus infected mails.
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > >
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> >
> > > > > > > >     >> >>>
> > > > > > > >     >> >
> > > > > > > >     >> >abhinandan.prateek@shapeblue.com
> > > > > > > >     >> >www.shapeblue.com
> > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > @shapeblue
> > > > > > > >     >> >
> > > > > > > >     >> >
> > > > > > > >     >> >
> > > > > > > >     >>
> > > > > > > >     >> abhinandan.prateek@shapeblue.com
> > > > > > > >     >> www.shapeblue.com
> > > > > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > >     >> @shapeblue
> > > > > > > >     >>
> > > > > > > >     >>
> > > > > > > >     >>
> > > > > > > >     >>
> > > > > > > >     >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > daan.hoogland@shapeblue.com
> > > > > > > > www.shapeblue.com
> > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > > @shapeblue
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Rafael Weingärtner
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Rafael Weingärtner
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Rafael Weingärtner
>

Re: Modern template hosting

Posted by Rafael Weingärtner <ra...@gmail.com>.

I think so far the mirror scheme we are discussing is not for general
purpose VMs, right? I was under the impression that we wanted to improve
the system VMs template management scheme ONLY.

On Fri, Mar 3, 2017 at 1:27 PM, Chiradeep Vittal <ch...@gmail.com>
wrote:

> I do feel like this is early optimization. Mirrors rarely fail. I'd expect
> a single web server hosted on Apache Infra without any monitors to fail
> more often than a mirror. We already expect Wido's systemvm repository to
> be up all the time. And it has been. Similarly, I don't believe Nux's
> repository has ever been down. And if Accelerite wants to host on S3, that
> one is pretty solid as well.
>
> This is an infrequent operation in a cloud. After the cloud is installed,
> the download servers are only needed for a new zone. If we trust the user
> to run a cloud, surely he/she can run a web server to serve some built-in
> templates. And if her cloud is successful, she needs to figure out how to
> host her templates anyway and not rely on 3rd parties.
>
>
> On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <ws...@cloudops.com>
> wrote:
>
> > 1) If the legacy implementations do not support redirects, that does
> cause
> > a problem.  A potential solution in that case is to have the web server
> > actually proxy the download, but that is not ideal and I would like to
> > avoid it if possible.  Thanks for bringing that up Chiradeep.
> >
> > 2) I think we need to have a single URL which people can target.  Once
> they
> > make the switch to the new URL, we want the implementation to be able to
> > handle mirror failures without affecting the end client.  We want to
> avoid
> > the situation where an ACS user will ever have to change this URL more
> than
> > once.  Mirror failures SHOULD NOT affect the ACS users assuming there is
> > still at least one mirror who can serve the requested resource.
> >
> > These are obviously my personal opinions and others will probably have
> > differing opinions.
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal <ch...@gmail.com>
> > wrote:
> >
> > > 1. If you are targeting legacy installations, they are not able to
> follow
> > > redirects. The line of code that added this capability was added on
> > 11/16.
> > > 2. If you trust the users to edit the database to change the URL, you
> can
> > > trust them to change it to anything. Just document a known list of good
> > > template locations.
> > >
> > > On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <ws...@cloudops.com>
> > > wrote:
> > >
> > > > So the main issue I see with this is the following.
> > > >
> > > > This implementation is designed to target legacy installations which
> > will
> > > > be affected when download.cloud.com disappears.  These people will
> > need
> > > to
> > > > make a DB change to replace the 'download.cloud.com' with some other
> > url
> > > > (maybe 'download.cloudstack.org').  Once the DB has been updated, we
> > can
> > > > not expect anything else of the client, they need to be able to just
> > > > continue operation as they were without any need to rebuild or
> upgrade
> > > > their ACS.
> > > >
> > > > If we try to force the decision to the client, then we break the
> legacy
> > > > implementations.  Since the SSVM is likely going to be the client in
> > some
> > > > cases, and since it does not already have the logic to handle the 300
> > > > approach correctly, I feel like it is not a viable solution for the
> > > legacy
> > > > deployments.
> > > >
> > > > In addition to that.  I don't think the client is in any better
> > position
> > > to
> > > > make the mirror decision than I am.  I am likely able to make a more
> > > > 'educated' decision than the client would be able to because I can do
> > > > different tests on the endpoint(s) before making a final decision.
> > > >
> > > > Does this make sense or am I missing something obvious?
> > > >
> > > > *Will STEVENS*
> > > > Lead Developer
> > > >
> > > > <https://goo.gl/NYZ8KK>
> > > >
> > > > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
> > > > rafaelweingartner@gmail.com> wrote:
> > > >
> > > > > I think I understood you, but I did not understand you.
> > > > >
> > > > > Let me see if I can create an example to illustrate how we could do
> > > using
> > > > > the HTTP 300 code.
> > > > >
> > > > > The HTTP 300 code indicates clients that he/she/it has multiple
> > choices
> > > > to
> > > > > access the requested resource. If the request is a HEAD, this is
> the
> > > only
> > > > > thing that the server returns. If the request if a GET, then the
> > server
> > > > > returns the choices for the client. The response type is defined on
> > the
> > > > > server based on the Content-Type, user agent and others.
> > > > >
> > > > > Let’s say the server receives a GET request and informs the
> > > content-type
> > > > as
> > > > > JSON (application/JSON). Your application could get the mirrors
> list,
> > > do
> > > > > the appending process for the requested resource path, and then
> > return
> > > a
> > > > > JSON array with the possible mirror the client can use. Then, the
> > > client
> > > > > gets this list and does the processing required to select a mirror.
> > > > >
> > > > > Now let’s say that a human access the link using a browser. Then,
> the
> > > > > server should receive something like (text/plain) as the
> > content-type;
> > > I
> > > > > think depending on the browser this field may differ a little (it
> > would
> > > > > require some checking). Anyways, the server detects that it is a
> > > “human”
> > > > > requesting the resource, then we could serve an HTML page with a
> > > > Javascript
> > > > > that uses the mirror list. This Javascript could do some checking
> and
> > > > > choose the best mirror for that specific.
> > > > >
> > > > > Why do I think this approach is interesting?
> > > > > The application you developed would be used only to retrieve valid
> > and
> > > > > trusted sources of system VM images (a trusted repository for
> mirrors
> > > > that
> > > > > we as a community vouch for).  Also, we shift the decision process
> > > > > regarding mirrors from the server to the client. Then, it is up to
> > > > clients
> > > > > to select mirrors, and not up to us (or some of our
> implementations).
> > > > >
> > > > > Did this help?
> > > > > BTW: I am not saying I am against the way you proposed, which would
> > > work
> > > > > fine. It is merely a suggestion using a different perspective.
> > > > >
> > > > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
> > > williamstevens@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > @rafael: in general, I don't think the client should ever be
> > making a
> > > > > > choice about a mirror. We have to assume we are working with a
> > > scripted
> > > > > > application and anything hard coded in that implementation on the
> > > > client
> > > > > > side is a risk.
> > > > > >
> > > > > > I may not be understanding the use of the 300 approach, so I will
> > > > > research
> > > > > > it to see if I can make it fit.
> > > > > >
> > > > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
> > > > > rafaelweingartner@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > Will great job.
> > > > > >
> > > > > > I had the same doubt as Daan.
> > > > > >
> > > > > > Have you considered using HTTP 300 working mode? Then, we could
> let
> > > the
> > > > > > client decide which mirror is the best(closest?). Section
> “10.3.1”
> > of
> > > > [1]
> > > > > > talks about it; this would put the pressure on deciding upon a
> > mirror
> > > > on
> > > > > > the client side, but I think it would be fair.
> > > > > >
> > > > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
> > > > > >
> > > > > > PS: You really like very short variables names!
> > > > > >
> > > > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
> > > > > daan.hoogland@shapeblue.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Nice little thing Will,
> > > > > > >
> > > > > > > One question: if I read the code correctly it ‘go’es and tries
> > all
> > > > > > mirrors
> > > > > > > at once and whichever responses first is redirected to the
> > client.
> > > > This
> > > > > > > might well be the same every time. This might be the one
> closest
> > to
> > > > the
> > > > > > > mirror302 and not to the client. Is that correct or did I miss
> a
> > > > > weighing
> > > > > > > algorithm hidden in there?
> > > > > > >
> > > > > > > Good coding,
> > > > > > >
> > > > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
> > > > > > >
> > > > > > >     Hey All,
> > > > > > >     Please review this repo and tell me what you think:
> > > > > > >     https://github.com/swill/mirror302
> > > > > > >
> > > > > > >     Let me know if you have questions or you would like me to
> > make
> > > > > > changes
> > > > > > >     before I bring this topic up with ASF Infra and open a
> ticket
> > > to
> > > > > get
> > > > > > > this
> > > > > > >     implemented.
> > > > > > >
> > > > > > >     Cheers,
> > > > > > >
> > > > > > >     *Will Stevens*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> > > > > wstevens@cloudops.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >     > I am building a short term solution right now.  I
> hopefully
> > > > will
> > > > > > > have the
> > > > > > >     > building blocks in place by the weekend so we can start
> > > working
> > > > > > with
> > > > > > > Infra
> > > > > > >     > to get it in place.
> > > > > > >     >
> > > > > > >     > We will have a web server which we can point something
> like
> > > > > > >     > downloads.cloudstack.org (or whatever url) at.  Then we
> > will
> > > > > have
> > > > > > > an ASF
> > > > > > >     > repo which tracks a mirror list and exposes a static site
> > to
> > > be
> > > > > > able
> > > > > > > to
> > > > > > >     > query the mirror list.
> > > > > > >     >
> > > > > > >     > The web server will receive a request and will do a 302
> > > > redirect
> > > > > to
> > > > > > > the
> > > > > > >     > appropriate resource on one of the mirrors.
> > > > > > >     >
> > > > > > >     > This gets us started.  Legacy environments will have to
> do
> > a
> > > DB
> > > > > > > change to
> > > > > > >     > change from 'download.cloud.com' to '
> > > downloads.cloudstack.org'
> > > > > (or
> > > > > > >     > whatever), but otherwise it should be pretty simple.
> > > > > > >     >
> > > > > > >     > We can improve how we deliver templates going forward,
> but
> > > this
> > > > > > > obviously
> > > > > > >     > requires some discussion still.  I will try to buy us
> some
> > > time
> > > > > > with
> > > > > > > an
> > > > > > >     > implementation which solves our problems today...
> > > > > > >     >
> > > > > > >     > *Will STEVENS*
> > > > > > >     > Lead Developer
> > > > > > >     >
> > > > > > >     > <https://goo.gl/NYZ8KK>
> > > > > > >     >
> > > > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > > > > > >     > abhinandan.prateek@shapeblue.com> wrote:
> > > > > > >     >
> > > > > > >     >>
> > > > > > >     >> Initial seeding is a manual step and that is not going
> to
> > > > change
> > > > > > in
> > > > > > > near
> > > > > > >     >> future. A handy list of official places from where these
> > > > > templates
> > > > > > > can be
> > > > > > >     >> downloaded won’t harm this part of the installation.
> Just
> > > > allow
> > > > > > > anyone
> > > > > > >     >> credible who follows democratic process to publish their
> > > > > templates
> > > > > > > with
> > > > > > >     >> apache’s blessings.
> > > > > > >     >>
> > > > > > >     >>
> > > > > > >     >> Coming to urls hard coded in db, the popular suggestion
> is
> > > to
> > > > > > point
> > > > > > > them
> > > > > > >     >> to mirrors. This looks good but this is going to take
> some
> > > > > effort
> > > > > > > that has
> > > > > > >     >> not materialised yet. Once this change is made it will
> be
> > > > going
> > > > > in
> > > > > > > some
> > > > > > >     >> future release like 4.11; the problem still remains for
> > the
> > > > > > > releases that
> > > > > > >     >> are already out there. In case access to
> > download.cloud.com
> > > > is
> > > > > > > dropped
> > > > > > >     >> at some point we will have to document the procedure
> such
> > > that
> > > > > > > people know
> > > > > > >     >> how to make these stable releases work. That is where
> the
> > > > > > procedure
> > > > > > >     >> documented here https://shankerbalan.net/blog/
> > > > > > >     >> seed-cloudstack-templates-offline/ should be officially
> > > > adapted
> > > > > > > with a
> > > > > > >     >> marketplace to pick up templates of choosing. Maybe we
> can
> > > > have
> > > > > a
> > > > > > > pre-setup
> > > > > > >     >> script to make choices instead of written wiki. Note
> that
> > > this
> > > > > > will
> > > > > > > be
> > > > > > >     >> anyway needed by the older releases.
> > > > > > >     >>
> > > > > > >     >>
> > > > > > >     >>
> > > > > > >     >>
> > > > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
> > > paul.angus@shapeblue.com>
> > > > > > > wrote:
> > > > > > >     >>
> > > > > > >     >> >-1
> > > > > > >     >> >
> > > > > > >     >> >We are trying to get MORE people to use CloudStack.
> And
> > > > > raising
> > > > > > > the
> > > > > > >     >> barrier to entry does nothing to help that.
> > > > > > >     >> >
> > > > > > >     >> >
> > > > > > >     >> >Kind regards,
> > > > > > >     >> >
> > > > > > >     >> >Paul Angus
> > > > > > >     >> >
> > > > > > >     >> >paul.angus@shapeblue.com
> > > > > > >     >> >www.shapeblue.com
> > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > >     >> >@shapeblue
> > > > > > >     >> >
> > > > > > >     >> >
> > > > > > >     >> >
> > > > > > >     >> >
> > > > > > >     >> >-----Original Message-----
> > > > > > >     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@
> > > > > > shapeblue.com]
> > > > > > >     >> >Sent: 01 March 2017 09:39
> > > > > > >     >> >To: dev@cloudstack.apache.org
> > > > > > >     >> >Subject: Re: Modern template hosting
> > > > > > >     >> >
> > > > > > >     >> >+1
> > > > > > >     >> >
> > > > > > >     >> >After seeing this discussion running in circles several
> > > > times,
> > > > > I
> > > > > > > think
> > > > > > >     >> we should at least get started with the simplest option.
> > > > > > >     >> >
> > > > > > >     >> >
> > > > > > >     >> >
> > > > > > >     >> >
> > > > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
> > > > > chiradeepv@gmail.com>
> > > > > > > wrote:
> > > > > > >     >> >
> > > > > > >     >> >>The alternative foolproof way is to simply not provide
> > the
> > > > > > > automatic
> > > > > > >     >> >>download.
> > > > > > >     >> >>
> > > > > > >     >> >>Just document the mirror list URL where the templates
> > can
> > > be
> > > > > > > found. I
> > > > > > >     >> >>don't know why people reject this option.
> > > > > > >     >> >>
> > > > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
> > > > > > templates-offline/
> > > > > > >     >> >>
> > > > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> > > > > > >     >> >><wi...@gmail.com>
> > > > > > >     >> >>wrote:
> > > > > > >     >> >>
> > > > > > >     >> >>> OK. Thanks for the heads up.
> > > > > > >     >> >>>
> > > > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > > > > chiradeepv@gmail.com>
> > > > > > >     >> wrote:
> > > > > > >     >> >>>
> > > > > > >     >> >>> > Sounds workable. The downloader code in the SSVM
> > won't
> > > > > > follow
> > > > > > >     >> >>> > redirects I think.
> > > > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
> > > > > > 5511065fc20787619d
> > > > > > > 9cd0444
> > > > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/
> > storage/template/
> > > > > > >     >> >>> > HttpTemplateDownloader.java#L93
> > > > > > >     >> >>> > https://goo.gl/dSi0r5
> > > > > > >     >> >>> >  Might need to add
> > > > > > >     >> >>> > client.setRedirectStrategy(new
> > LaxRedirectStrategy());
> > > > > > >     >> >>> >
> > > > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> > > > > > >     >> >>> > <ws...@cloudops.com>
> > > > > > >     >> >>> > wrote:
> > > > > > >     >> >>> >
> > > > > > >     >> >>> > > We haven't opened a ticket yet because we don't
> > > have a
> > > > > > > strategy
> > > > > > >     >> yet.
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> > > What do you guys think of this:
> > > > > > >     >> >>> > > - We setup a new github repo in the 'apache' org
> > > which
> > > > > > > consists
> > > > > > >     >> >>> > > of a
> > > > > > >     >> >>> > single
> > > > > > >     >> >>> > > file with a list of active/supported mirrors.
> > > > > > >     >> >>> > > - I write a small web server, distributed as a
> > > binary,
> > > > > > > which can
> > > > > > >     >> >>> > > be
> > > > > > >     >> >>> > hosted
> > > > > > >     >> >>> > > by ASF Infra.  This web server will query the
> > > current
> > > > > list
> > > > > > > of
> > > > > > >     >> >>> > > mirrors
> > > > > > >     >> >>> and
> > > > > > >     >> >>> > > will select one and then do a 302 redirect to
> that
> > > > > mirror.
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> > > The act of 'choosing' a mirror could be done in
> a
> > > > number
> > > > > > of
> > > > > > > ways.
> > > > > > >     >> >>> > > - If we want to define an order, then it could
> > just
> > > > try
> > > > > > > from the
> > > > > > >     >> >>> > > top of
> > > > > > >     >> >>> > the
> > > > > > >     >> >>> > > list and work its way down.  It would curl the
> > > target
> > > > to
> > > > > > > make
> > > > > > >     >> >>> > > sure it
> > > > > > >     >> >>> > gets
> > > > > > >     >> >>> > > a 200 and if it does, it would do a 302
> redirect.
> > > > > > >     >> >>> > > - Or, if we want to distribute the load across
> the
> > > > > > mirrors,
> > > > > > > we
> > > > > > >     >> >>> > > could
> > > > > > >     >> >>> pick
> > > > > > >     >> >>> > > from the list randomly.  Again, doing a curl to
> > > verify
> > > > > the
> > > > > > > mirror
> > > > > > >     >> >>> > > is up
> > > > > > >     >> >>> > and
> > > > > > >     >> >>> > > then doing a redirect.
> > > > > > >     >> >>> > > - If we want to get fancy, we could do a reverse
> > IP
> > > > > lookup
> > > > > > > and
> > > > > > >     >> >>> > > try to
> > > > > > >     >> >>> > match
> > > > > > >     >> >>> > > the requester with their closest geographical
> > > mirror.
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> > > Thoughts?
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> > > *Will STEVENS*
> > > > > > >     >> >>> > > Lead Developer
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep
> > Vittal <
> > > > > > >     >> >>> chiradeepv@gmail.com
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> > > wrote:
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> > > > What steps are needed to set up a mirror? What
> > > does
> > > > > > Infra
> > > > > > > need
> > > > > > >     >> to do?
> > > > > > >     >> >>> > Has
> > > > > > >     >> >>> > > > anybody filed a ticket with Infra?
> > > > > > >     >> >>> > > >
> > > > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja
> Pullela <
> > > > > > >     >> >>> > > > raja.pullela@accelerite.com>
> > > > > > >     >> >>> > > > wrote:
> > > > > > >     >> >>> > > >
> > > > > > >     >> >>> > > > > Hi will,
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > I believe, we didn’t get to close ‘getting a
> > > > mirror
> > > > > on
> > > > > > > Apache’
> > > > > > >     >> >>> > because
> > > > > > >     >> >>> > > we
> > > > > > >     >> >>> > > > > needed someone on the Apache Infra side to
> > close
> > > > > this.
> > > > > > > BTW,
> > > > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages
> > > this?)
> > > > > has
> > > > > > >     >> >>> > > > > all/most of
> > > > > > >     >> >>> > the
> > > > > > >     >> >>> > > > > content.  Once we can close on the Apache
> > mirror
> > > > for
> > > > > > > hosting
> > > > > > >     >> >>> > > > > the
> > > > > > >     >> >>> > > > content, I
> > > > > > >     >> >>> > > > > can help assist getting the content there.
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > For now, we have replicated the
> > > > download.cloud.com
> > > > > > > content to
> > > > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
> > > > > > >     >> >>> > > > > Also, we are working on a set of
> > steps/procedure
> > > > to
> > > > > > > help with
> > > > > > >     >> >>> > > > > this change.  I will update everyone in
> about
> > a
> > > > > week’s
> > > > > > > time
> > > > > > >     >> >>> > > > > on the
> > > > > > >     >> >>> > details.
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > Best,
> > > > > > >     >> >>> > > > > Raja Pullela
> > > > > > >     >> >>> > > > > Engineering Team,
> > > > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa
> Clara,
> > > CA,
> > > > > > 95054
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "
> > williamstevens@gmail.com
> > > > on
> > > > > > > behalf of
> > > > > > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com on
> > > behalf
> > > > > of
> > > > > > >     >> >>> > wstevens@cloudops.com>
> > > > > > >     >> >>> > > > > wrote:
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > unfortunately the template mirror
> conversation
> > > got
> > > > > > > caught up
> > > > > > >     >> >>> > > > > in
> > > > > > >     >> >>> > details
> > > > > > >     >> >>> > > > and
> > > > > > >     >> >>> > > > > nobody took the lead on implementing a
> > solution.
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > citrix has been pinging me every couple
> months
> > > to
> > > > > say
> > > > > > > 'dude,
> > > > > > >     >> >>> > > > > we
> > > > > > >     >> >>> need
> > > > > > >     >> >>> > to
> > > > > > >     >> >>> > > > > remove the dependency on
> download.citrix.com
> > ',
> > > > but
> > > > > i
> > > > > > > have not
> > > > > > >     >> >>> > > > > had
> > > > > > >     >> >>> > the
> > > > > > >     >> >>> > > > > cycles to get in and solve the problem.  the
> > > > > shutdown
> > > > > > > of that
> > > > > > >     >> >>> > > > > is
> > > > > > >     >> >>> > > imminent
> > > > > > >     >> >>> > > > > right now, so we need to solve it asap.
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > *Will STEVENS*
> > > > > > >     >> >>> > > > > Lead Developer
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul
> Angus <
> > > > > > >     >> >>> > paul.angus@shapeblue.com
> > > > > > >     >> >>> > > >
> > > > > > >     >> >>> > > > > wrote:
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > > Hi Nathan,
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > Ideally, if you put the template location
> in
> > > (or
> > > > > use
> > > > > > a
> > > > > > >     >> >>> > > > > > template
> > > > > > >     >> >>> > > defined
> > > > > > >     >> >>> > > > > > in)  test_data.py then the actual location
> > can
> > > > be
> > > > > > >     >> >>> > > > > > overridden by
> > > > > > >     >> >>> > > anyone
> > > > > > >     >> >>> > > > > > testing.
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > For Trillian, we've copied all of the
> > > templates
> > > > > that
> > > > > > > people
> > > > > > >     >> >>> > > > > > have
> > > > > > >     >> >>> > > define
> > > > > > >     >> >>> > > > > to
> > > > > > >     >> >>> > > > > > a local repo and then replace the URLs in
> > > > > > > test_data.py to
> > > > > > >     >> >>> > > > > > reduce
> > > > > > >     >> >>> > > > > bandwidth
> > > > > > >     >> >>> > > > > > use and download times.
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > Ie:
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >             "bootableIso":
> > > > > > >     >> >>> > > > > >                 {
> > > > > > >     >> >>> > > > > >                     "displaytext": "Test
> > > > Bootable
> > > > > > > ISO",
> > > > > > >     >> >>> > > > > >                     "name": "testISO",
> > > > > > >     >> >>> > > > > >                     "bootable": True,
> > > > > > >     >> >>> > > > > >                     "ispublic": False,
> > > > > > >     >> >>> > > > > >                     "url": "{{
> > > > > > marvin_images_location
> > > > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
> > > > > > >     >> >>> > > > > >                     "ostype": 'Other Linux
> > > > > > (64-bit)',
> > > > > > >     >> >>> > > > > >                     "mode":
> 'HTTP_DOWNLOAD'
> > > > > > >     >> >>> > > > > >         },
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > I thought that we had come up with a
> > solution
> > > > for
> > > > > > >     >> >>> > download.cloud.com
> > > > > > >     >> >>> > > ,
> > > > > > >     >> >>> > > > by
> > > > > > >     >> >>> > > > > > having a mirrorlist hosted in Community
> > Apache
> > > > > > > 'space' with
> > > > > > >     >> >>> anyone
> > > > > > >     >> >>> > > able
> > > > > > >     >> >>> > > > > to
> > > > > > >     >> >>> > > > > > out themselves forward as a mirror.
> > > > > > >     >> >>> > > > > > But I must admit I lost track of whether
> > > anyone
> > > > > made
> > > > > > > the
> > > > > > >     >> >>> requisite
> > > > > > >     >> >>> > > > > changes
> > > > > > >     >> >>> > > > > > in code....
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > Kind regards,
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > Paul Angus
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > paul.angus@shapeblue.com
> > > > > > >     >> >>> > > > > > www.shapeblue.com
> > > > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden, London
> > WC2N
> > > > > 4HSUK
> > > > > > >     >> >>> > > > > > @shapeblue
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > -----Original Message-----
> > > > > > >     >> >>> > > > > > From: williamstevens@gmail.com
> > > > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
> > > > > > >     >> >>> > On
> > > > > > >     >> >>> > > > > > Behalf Of Will Stevens
> > > > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
> > > > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
> > > > > > >     >> >>> > > > > > Subject: Re: Modern template hosting
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > this is a hard questions.  in general, we
> > > should
> > > > > be
> > > > > > > setting
> > > > > > >     >> >>> > > > > > up a
> > > > > > >     >> >>> > > mirror
> > > > > > >     >> >>> > > > > on
> > > > > > >     >> >>> > > > > > some cloudstack/apache domain and then
> > mirror
> > > to
> > > > > > other
> > > > > > >     >> >>> > > > > > provided
> > > > > > >     >> >>> > > > > templates.
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > we MUST come up with a solution to
> deprecate
> > > > > > >     >> >>> > > > > > 'download.cloud.com
> > > > > > >     >> >>> ',
> > > > > > >     >> >>> > > > that
> > > > > > >     >> >>> > > > > > is going to be going away any day now.
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > i don't know the right way to solve this
> to
> > be
> > > > > > > honest, but
> > > > > > >     >> >>> > > > > > if you
> > > > > > >     >> >>> > > have
> > > > > > >     >> >>> > > > > > ideas, i am willing to help.
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > *Will STEVENS*
> > > > > > >     >> >>> > > > > > Lead Developer
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan
> > > > Johnson <
> > > > > > >     >> >>> njohnson@ena.com
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> > > > > wrote:
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > > So not to re-open a can of worms, but
> I’m
> > > in a
> > > > > > > situation
> > > > > > >     >> >>> > > > > > > where
> > > > > > >     >> >>> I
> > > > > > >     >> >>> > > need
> > > > > > >     >> >>> > > > > > > to come up with a Marvin component test
> > that
> > > > > > > depends on a
> > > > > > >     >> >>> > template
> > > > > > >     >> >>> > > > > > > based on a kernel that’s relatively new,
> > > i.e.,
> > > > > > > newer than
> > > > > > >     >> >>> Centos
> > > > > > >     >> >>> > > 5.3
> > > > > > >     >> >>> > > > /
> > > > > > >     >> >>> > > > > > Ubuntu 10.04 .
> > > > > > >     >> >>> > > > > > > I see openvm.eu has a suitable template
> > > > (Ubuntu
> > > > > > > 16.0.4
> > > > > > >     >> >>> > > > > > > for
> > > > > > >     >> >>> KVM),
> > > > > > >     >> >>> > > but
> > > > > > >     >> >>> > > > > > > from looking at the thread "Migrating
> > > > CloudStack
> > > > > > > content
> > > > > > >     >> >>> > > > > > > from download.cloud.com” it looks like
> > > there
> > > > is
> > > > > > >     >> >>> > > > > > > resistance to using
> > > > > > >     >> >>> > > this
> > > > > > >     >> >>> > > > at
> > > > > > >     >> >>> > > > > > > least for hosting system vm templates
> over
> > > > > > concerns
> > > > > > > of
> > > > > > >     >> >>> > neutrality.
> > > > > > >     >> >>> > > > > > > Would this be suitable for a component
> > test?
> > > > If
> > > > > > > not,
> > > > > > >     >> >>> > > > > > > what is a
> > > > > > >     >> >>> > > > > “blessed”
> > > > > > >     >> >>> > > > > > > template location?
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > > Thanks in advance!
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > > Nathan Johnson
> > > > > > >     >> >>> > > > > > > R&D Engineer
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12
> > > Nashville,
> > > > TN
> > > > > > > 37211
> > > > > > >     >> >>> > > > > > > General Office: 615-312-6000
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > > website | blog | support
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > DISCLAIMER
> > > > > > >     >> >>> > > > > ==========
> > > > > > >     >> >>> > > > > This e-mail may contain privileged and
> > > > confidential
> > > > > > >     >> >>> > > > > information
> > > > > > >     >> >>> which
> > > > > > >     >> >>> > > is
> > > > > > >     >> >>> > > > > the property of Accelerite, a Persistent
> > Systems
> > > > > > > business. It
> > > > > > >     >> >>> > > > > is
> > > > > > >     >> >>> > > intended
> > > > > > >     >> >>> > > > > only for the use of the individual or entity
> > to
> > > > > which
> > > > > > > it is
> > > > > > >     >> >>> > addressed.
> > > > > > >     >> >>> > > If
> > > > > > >     >> >>> > > > > you are not the intended recipient, you are
> > not
> > > > > > > authorized to
> > > > > > >     >> >>> > > > > read,
> > > > > > >     >> >>> > > > retain,
> > > > > > >     >> >>> > > > > copy, print, distribute or use this message.
> > If
> > > > you
> > > > > > have
> > > > > > >     >> >>> > > > > received
> > > > > > >     >> >>> > this
> > > > > > >     >> >>> > > > > communication in error, please notify the
> > sender
> > > > and
> > > > > > > delete
> > > > > > >     >> >>> > > > > all
> > > > > > >     >> >>> > copies
> > > > > > >     >> >>> > > of
> > > > > > >     >> >>> > > > > this message. Accelerite, a Persistent
> Systems
> > > > > > business
> > > > > > > does
> > > > > > >     >> >>> > > > > not
> > > > > > >     >> >>> > accept
> > > > > > >     >> >>> > > > any
> > > > > > >     >> >>> > > > > liability for virus infected mails.
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > >
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> >
> > > > > > >     >> >>>
> > > > > > >     >> >
> > > > > > >     >> >abhinandan.prateek@shapeblue.com
> > > > > > >     >> >www.shapeblue.com
> > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > @shapeblue
> > > > > > >     >> >
> > > > > > >     >> >
> > > > > > >     >> >
> > > > > > >     >>
> > > > > > >     >> abhinandan.prateek@shapeblue.com
> > > > > > >     >> www.shapeblue.com
> > > > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > >     >> @shapeblue
> > > > > > >     >>
> > > > > > >     >>
> > > > > > >     >>
> > > > > > >     >>
> > > > > > >     >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > daan.hoogland@shapeblue.com
> > > > > > > www.shapeblue.com
> > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > @shapeblue
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Rafael Weingärtner
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rafael Weingärtner
> > > > >
> > > >
> > >
> >
>



-- 
Rafael Weingärtner

Re: Modern template hosting

Posted by Will Stevens <wi...@gmail.com>.

I don't think your analogy works. It is difficult to get acs up and
running. You may take it for granted, but I know that struggle has made
some people move on.

I think it is more analogous with not buying a car because the dealer won't
give you the keys until you can prove you can change the oil.

On Mar 3, 2017 6:36 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:

> To me, that is like saying that a customer won't buy a car because he can't
> figure out how to roll down his window, so let's build a voice-controlled
> window controller. Can be done, sounds great, but what happens when that
> voice controller thing in the cloud goes down?
>
>
> On Fri, Mar 3, 2017 at 3:09 PM, Will Stevens <wi...@gmail.com>
> wrote:
>
> > I agree with Paul. Look at the list of things they have to
> > learn/master/care about. We don't want to add to that list, we want to
> > remove from that list. Make the project/product more accessible. Reduce
> the
> > barrier to entry...
> >
> > On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:
> >
> > > Seriously?
> > > apt-get install apache2
> > >
> > > To install CloudStack, they need to know
> > > - DB installation, perhaps some SQL
> > > - VLAN configuration on their switches
> > > - Ins-and-outs between port forwarding, static ips,
> > > - NFS
> > > - package management
> > > - VHDs, qcow2. vmdk
> > > - hyperviosrs
> > > - and on and on.
> > >
> > > If they can figure all that out and not figure out a web server, they
> can
> > > always come to the mailing list.
> > >
> > >
> > >
> > >
> > > On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <pa...@shapeblue.com>
> > > wrote:
> > >
> > > > The issue is not with supporting highly experienced cloud operators.
> It
> > > is
> > > > an issue for new users and other 'relatively' inexperienced
> operators.
> > > >
> > > > I have helped enough newbies and cloud operators who have been
> running
> > > > their cloud for a while to know that the barriers to entry are too
> high
> > > as
> > > > they are. And telling anyone that they need to create a web server so
> > > that
> > > > they can add their initial template to get started or in order to
> > create
> > > a
> > > > new zone, just isn't going to fly.
> > > >
> > > > I'm a huge advocate of the 'download.cloudstack.org' endpoint which
> > the
> > > > community can add/remove mirrors to or from for system VMs or
> built-in
> > > > templates.  Can the same system be used for binary repos ? although
> > there
> > > > is an added complication of redist vs no-redist there...
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > paul.angus@shapeblue.com
> > > > www.shapeblue.com
> > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > @shapeblue
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Chiradeep Vittal [mailto:chiradeepv@gmail.com]
> > > > Sent: 03 March 2017 18:28
> > > > To: dev <de...@cloudstack.apache.org>
> > > > Subject: Re: Modern template hosting
> > > >
> > > > I do feel like this is early optimization. Mirrors rarely fail. I'd
> > > expect
> > > > a single web server hosted on Apache Infra without any monitors to
> fail
> > > > more often than a mirror. We already expect Wido's systemvm
> repository
> > to
> > > > be up all the time. And it has been. Similarly, I don't believe Nux's
> > > > repository has ever been down. And if Accelerite wants to host on S3,
> > > that
> > > > one is pretty solid as well.
> > > >
> > > > This is an infrequent operation in a cloud. After the cloud is
> > installed,
> > > > the download servers are only needed for a new zone. If we trust the
> > user
> > > > to run a cloud, surely he/she can run a web server to serve some
> > built-in
> > > > templates. And if her cloud is successful, she needs to figure out
> how
> > to
> > > > host her templates anyway and not rely on 3rd parties.
> > > >
> > > >
> > > > On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <ws...@cloudops.com>
> > > > wrote:
> > > >
> > > > > 1) If the legacy implementations do not support redirects, that
> does
> > > > > cause a problem.  A potential solution in that case is to have the
> > web
> > > > > server actually proxy the download, but that is not ideal and I
> would
> > > > > like to avoid it if possible.  Thanks for bringing that up
> Chiradeep.
> > > > >
> > > > > 2) I think we need to have a single URL which people can target.
> > Once
> > > > > they make the switch to the new URL, we want the implementation to
> be
> > > > > able to handle mirror failures without affecting the end client.
> We
> > > > > want to avoid the situation where an ACS user will ever have to
> > change
> > > > > this URL more than once.  Mirror failures SHOULD NOT affect the ACS
> > > > > users assuming there is still at least one mirror who can serve the
> > > > requested resource.
> > > > >
> > > > > These are obviously my personal opinions and others will probably
> > have
> > > > > differing opinions.
> > > > >
> > > > > *Will STEVENS*
> > > > > Lead Developer
> > > > >
> > > > > <https://goo.gl/NYZ8KK>
> > > > >
> > > > > On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal
> > > > > <ch...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > 1. If you are targeting legacy installations, they are not able
> to
> > > > > > follow redirects. The line of code that added this capability was
> > > > > > added on
> > > > > 11/16.
> > > > > > 2. If you trust the users to edit the database to change the URL,
> > > > > > you can trust them to change it to anything. Just document a
> known
> > > > > > list of good template locations.
> > > > > >
> > > > > > On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <
> > wstevens@cloudops.com>
> > > > > > wrote:
> > > > > >
> > > > > > > So the main issue I see with this is the following.
> > > > > > >
> > > > > > > This implementation is designed to target legacy installations
> > > > > > > which
> > > > > will
> > > > > > > be affected when download.cloud.com disappears.  These people
> > will
> > > > > need
> > > > > > to
> > > > > > > make a DB change to replace the 'download.cloud.com' with some
> > > > > > > other
> > > > > url
> > > > > > > (maybe 'download.cloudstack.org').  Once the DB has been
> > updated,
> > > > > > > we
> > > > > can
> > > > > > > not expect anything else of the client, they need to be able to
> > > > > > > just continue operation as they were without any need to
> rebuild
> > > > > > > or upgrade their ACS.
> > > > > > >
> > > > > > > If we try to force the decision to the client, then we break
> the
> > > > > > > legacy implementations.  Since the SSVM is likely going to be
> the
> > > > > > > client in
> > > > > some
> > > > > > > cases, and since it does not already have the logic to handle
> the
> > > > > > > 300 approach correctly, I feel like it is not a viable solution
> > > > > > > for the
> > > > > > legacy
> > > > > > > deployments.
> > > > > > >
> > > > > > > In addition to that.  I don't think the client is in any better
> > > > > position
> > > > > > to
> > > > > > > make the mirror decision than I am.  I am likely able to make a
> > > > > > > more 'educated' decision than the client would be able to
> because
> > > > > > > I can do different tests on the endpoint(s) before making a
> final
> > > > decision.
> > > > > > >
> > > > > > > Does this make sense or am I missing something obvious?
> > > > > > >
> > > > > > > *Will STEVENS*
> > > > > > > Lead Developer
> > > > > > >
> > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > >
> > > > > > > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
> > > > > > > rafaelweingartner@gmail.com> wrote:
> > > > > > >
> > > > > > > > I think I understood you, but I did not understand you.
> > > > > > > >
> > > > > > > > Let me see if I can create an example to illustrate how we
> > could
> > > > > > > > do
> > > > > > using
> > > > > > > > the HTTP 300 code.
> > > > > > > >
> > > > > > > > The HTTP 300 code indicates clients that he/she/it has
> multiple
> > > > > choices
> > > > > > > to
> > > > > > > > access the requested resource. If the request is a HEAD, this
> > is
> > > > > > > > the
> > > > > > only
> > > > > > > > thing that the server returns. If the request if a GET, then
> > the
> > > > > server
> > > > > > > > returns the choices for the client. The response type is
> > defined
> > > > > > > > on
> > > > > the
> > > > > > > > server based on the Content-Type, user agent and others.
> > > > > > > >
> > > > > > > > Let’s say the server receives a GET request and informs the
> > > > > > content-type
> > > > > > > as
> > > > > > > > JSON (application/JSON). Your application could get the
> mirrors
> > > > > > > > list,
> > > > > > do
> > > > > > > > the appending process for the requested resource path, and
> then
> > > > > return
> > > > > > a
> > > > > > > > JSON array with the possible mirror the client can use. Then,
> > > > > > > > the
> > > > > > client
> > > > > > > > gets this list and does the processing required to select a
> > > mirror.
> > > > > > > >
> > > > > > > > Now let’s say that a human access the link using a browser.
> > > > > > > > Then, the server should receive something like (text/plain)
> as
> > > > > > > > the
> > > > > content-type;
> > > > > > I
> > > > > > > > think depending on the browser this field may differ a little
> > > > > > > > (it
> > > > > would
> > > > > > > > require some checking). Anyways, the server detects that it
> is
> > a
> > > > > > “human”
> > > > > > > > requesting the resource, then we could serve an HTML page
> with
> > a
> > > > > > > Javascript
> > > > > > > > that uses the mirror list. This Javascript could do some
> > > > > > > > checking and choose the best mirror for that specific.
> > > > > > > >
> > > > > > > > Why do I think this approach is interesting?
> > > > > > > > The application you developed would be used only to retrieve
> > > > > > > > valid
> > > > > and
> > > > > > > > trusted sources of system VM images (a trusted repository for
> > > > > > > > mirrors
> > > > > > > that
> > > > > > > > we as a community vouch for).  Also, we shift the decision
> > > > > > > > process regarding mirrors from the server to the client.
> Then,
> > > > > > > > it is up to
> > > > > > > clients
> > > > > > > > to select mirrors, and not up to us (or some of our
> > > > implementations).
> > > > > > > >
> > > > > > > > Did this help?
> > > > > > > > BTW: I am not saying I am against the way you proposed, which
> > > > > > > > would
> > > > > > work
> > > > > > > > fine. It is merely a suggestion using a different
> perspective.
> > > > > > > >
> > > > > > > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
> > > > > > williamstevens@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > @rafael: in general, I don't think the client should ever
> be
> > > > > making a
> > > > > > > > > choice about a mirror. We have to assume we are working
> with
> > a
> > > > > > scripted
> > > > > > > > > application and anything hard coded in that implementation
> on
> > > > > > > > > the
> > > > > > > client
> > > > > > > > > side is a risk.
> > > > > > > > >
> > > > > > > > > I may not be understanding the use of the 300 approach, so
> I
> > > > > > > > > will
> > > > > > > > research
> > > > > > > > > it to see if I can make it fit.
> > > > > > > > >
> > > > > > > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
> > > > > > > > rafaelweingartner@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Will great job.
> > > > > > > > >
> > > > > > > > > I had the same doubt as Daan.
> > > > > > > > >
> > > > > > > > > Have you considered using HTTP 300 working mode? Then, we
> > > > > > > > > could let
> > > > > > the
> > > > > > > > > client decide which mirror is the best(closest?). Section
> > > > “10.3.1”
> > > > > of
> > > > > > > [1]
> > > > > > > > > talks about it; this would put the pressure on deciding
> upon
> > a
> > > > > mirror
> > > > > > > on
> > > > > > > > > the client side, but I think it would be fair.
> > > > > > > > >
> > > > > > > > > [1] https://www.w3.org/Protocols/
> rfc2616/rfc2616-sec10.html
> > > > > > > > >
> > > > > > > > > PS: You really like very short variables names!
> > > > > > > > >
> > > > > > > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
> > > > > > > > daan.hoogland@shapeblue.com
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Nice little thing Will,
> > > > > > > > > >
> > > > > > > > > > One question: if I read the code correctly it ‘go’es and
> > > > > > > > > > tries
> > > > > all
> > > > > > > > > mirrors
> > > > > > > > > > at once and whichever responses first is redirected to
> the
> > > > > client.
> > > > > > > This
> > > > > > > > > > might well be the same every time. This might be the one
> > > > > > > > > > closest
> > > > > to
> > > > > > > the
> > > > > > > > > > mirror302 and not to the client. Is that correct or did I
> > > > > > > > > > miss a
> > > > > > > > weighing
> > > > > > > > > > algorithm hidden in there?
> > > > > > > > > >
> > > > > > > > > > Good coding,
> > > > > > > > > >
> > > > > > > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org>
> > wrote:
> > > > > > > > > >
> > > > > > > > > >     Hey All,
> > > > > > > > > >     Please review this repo and tell me what you think:
> > > > > > > > > >     https://github.com/swill/mirror302
> > > > > > > > > >
> > > > > > > > > >     Let me know if you have questions or you would like
> me
> > > > > > > > > > to
> > > > > make
> > > > > > > > > changes
> > > > > > > > > >     before I bring this topic up with ASF Infra and open
> a
> > > > > > > > > > ticket
> > > > > > to
> > > > > > > > get
> > > > > > > > > > this
> > > > > > > > > >     implemented.
> > > > > > > > > >
> > > > > > > > > >     Cheers,
> > > > > > > > > >
> > > > > > > > > >     *Will Stevens*
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> > > > > > > > wstevens@cloudops.com
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >     > I am building a short term solution right now.  I
> > > > > > > > > > hopefully
> > > > > > > will
> > > > > > > > > > have the
> > > > > > > > > >     > building blocks in place by the weekend so we can
> > > > > > > > > > start
> > > > > > working
> > > > > > > > > with
> > > > > > > > > > Infra
> > > > > > > > > >     > to get it in place.
> > > > > > > > > >     >
> > > > > > > > > >     > We will have a web server which we can point
> > something
> > > > like
> > > > > > > > > >     > downloads.cloudstack.org (or whatever url) at.
> Then
> > > > > > > > > > we
> > > > > will
> > > > > > > > have
> > > > > > > > > > an ASF
> > > > > > > > > >     > repo which tracks a mirror list and exposes a
> static
> > > > > > > > > > site
> > > > > to
> > > > > > be
> > > > > > > > > able
> > > > > > > > > > to
> > > > > > > > > >     > query the mirror list.
> > > > > > > > > >     >
> > > > > > > > > >     > The web server will receive a request and will do a
> > > > > > > > > > 302
> > > > > > > redirect
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > >     > appropriate resource on one of the mirrors.
> > > > > > > > > >     >
> > > > > > > > > >     > This gets us started.  Legacy environments will
> have
> > > > > > > > > > to do
> > > > > a
> > > > > > DB
> > > > > > > > > > change to
> > > > > > > > > >     > change from 'download.cloud.com' to '
> > > > > > downloads.cloudstack.org'
> > > > > > > > (or
> > > > > > > > > >     > whatever), but otherwise it should be pretty
> simple.
> > > > > > > > > >     >
> > > > > > > > > >     > We can improve how we deliver templates going
> > forward,
> > > > > > > > > > but
> > > > > > this
> > > > > > > > > > obviously
> > > > > > > > > >     > requires some discussion still.  I will try to buy
> us
> > > > > > > > > > some
> > > > > > time
> > > > > > > > > with
> > > > > > > > > > an
> > > > > > > > > >     > implementation which solves our problems today...
> > > > > > > > > >     >
> > > > > > > > > >     > *Will STEVENS*
> > > > > > > > > >     > Lead Developer
> > > > > > > > > >     >
> > > > > > > > > >     > <https://goo.gl/NYZ8KK>
> > > > > > > > > >     >
> > > > > > > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek
> <
> > > > > > > > > >     > abhinandan.prateek@shapeblue.com> wrote:
> > > > > > > > > >     >
> > > > > > > > > >     >>
> > > > > > > > > >     >> Initial seeding is a manual step and that is not
> > > > > > > > > > going to
> > > > > > > change
> > > > > > > > > in
> > > > > > > > > > near
> > > > > > > > > >     >> future. A handy list of official places from where
> > > > > > > > > > these
> > > > > > > > templates
> > > > > > > > > > can be
> > > > > > > > > >     >> downloaded won’t harm this part of the
> installation.
> > > > > > > > > > Just
> > > > > > > allow
> > > > > > > > > > anyone
> > > > > > > > > >     >> credible who follows democratic process to publish
> > > > > > > > > > their
> > > > > > > > templates
> > > > > > > > > > with
> > > > > > > > > >     >> apache’s blessings.
> > > > > > > > > >     >>
> > > > > > > > > >     >>
> > > > > > > > > >     >> Coming to urls hard coded in db, the popular
> > > > > > > > > > suggestion is
> > > > > > to
> > > > > > > > > point
> > > > > > > > > > them
> > > > > > > > > >     >> to mirrors. This looks good but this is going to
> > take
> > > > > > > > > > some
> > > > > > > > effort
> > > > > > > > > > that has
> > > > > > > > > >     >> not materialised yet. Once this change is made it
> > > > > > > > > > will be
> > > > > > > going
> > > > > > > > in
> > > > > > > > > > some
> > > > > > > > > >     >> future release like 4.11; the problem still
> remains
> > > > > > > > > > for
> > > > > the
> > > > > > > > > > releases that
> > > > > > > > > >     >> are already out there. In case access to
> > > > > download.cloud.com
> > > > > > > is
> > > > > > > > > > dropped
> > > > > > > > > >     >> at some point we will have to document the
> procedure
> > > > > > > > > > such
> > > > > > that
> > > > > > > > > > people know
> > > > > > > > > >     >> how to make these stable releases work. That is
> > where
> > > > > > > > > > the
> > > > > > > > > procedure
> > > > > > > > > >     >> documented here https://shankerbalan.net/blog/
> > > > > > > > > >     >> seed-cloudstack-templates-offline/ should be
> > > > > > > > > > officially
> > > > > > > adapted
> > > > > > > > > > with a
> > > > > > > > > >     >> marketplace to pick up templates of choosing.
> Maybe
> > > > > > > > > > we can
> > > > > > > have
> > > > > > > > a
> > > > > > > > > > pre-setup
> > > > > > > > > >     >> script to make choices instead of written wiki.
> Note
> > > > > > > > > > that
> > > > > > this
> > > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > >     >> anyway needed by the older releases.
> > > > > > > > > >     >>
> > > > > > > > > >     >>
> > > > > > > > > >     >>
> > > > > > > > > >     >>
> > > > > > > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
> > > > > > paul.angus@shapeblue.com>
> > > > > > > > > > wrote:
> > > > > > > > > >     >>
> > > > > > > > > >     >> >-1
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >We are trying to get MORE people to use
> CloudStack.
> > > > > > > > > > And
> > > > > > > > raising
> > > > > > > > > > the
> > > > > > > > > >     >> barrier to entry does nothing to help that.
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >Kind regards,
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >Paul Angus
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >paul.angus@shapeblue.com
> > > > > > > > > >     >> >www.shapeblue.com
> > > > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N
> 4HSUK
> > > > > > > > > >     >> >@shapeblue
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >-----Original Message-----
> > > > > > > > > >     >> >From: Abhinandan Prateek [mailto:
> > abhinandan.prateek@
> > > > > > > > > shapeblue.com]
> > > > > > > > > >     >> >Sent: 01 March 2017 09:39
> > > > > > > > > >     >> >To: dev@cloudstack.apache.org
> > > > > > > > > >     >> >Subject: Re: Modern template hosting
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >+1
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >After seeing this discussion running in circles
> > > > > > > > > > several
> > > > > > > times,
> > > > > > > > I
> > > > > > > > > > think
> > > > > > > > > >     >> we should at least get started with the simplest
> > > option.
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
> > > > > > > > chiradeepv@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >>The alternative foolproof way is to simply not
> > > > > > > > > > provide
> > > > > the
> > > > > > > > > > automatic
> > > > > > > > > >     >> >>download.
> > > > > > > > > >     >> >>
> > > > > > > > > >     >> >>Just document the mirror list URL where the
> > > > > > > > > > templates
> > > > > can
> > > > > > be
> > > > > > > > > > found. I
> > > > > > > > > >     >> >>don't know why people reject this option.
> > > > > > > > > >     >> >>
> > > > > > > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
> > > > > > > > > templates-offline/
> > > > > > > > > >     >> >>
> > > > > > > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> > > > > > > > > >     >> >><wi...@gmail.com>
> > > > > > > > > >     >> >>wrote:
> > > > > > > > > >     >> >>
> > > > > > > > > >     >> >>> OK. Thanks for the heads up.
> > > > > > > > > >     >> >>>
> > > > > > > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > > > > > > > chiradeepv@gmail.com>
> > > > > > > > > >     >> wrote:
> > > > > > > > > >     >> >>>
> > > > > > > > > >     >> >>> > Sounds workable. The downloader code in the
> > > > > > > > > > SSVM
> > > > > won't
> > > > > > > > > follow
> > > > > > > > > >     >> >>> > redirects I think.
> > > > > > > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
> > > > > > > > > 5511065fc20787619d
> > > > > > > > > > 9cd0444
> > > > > > > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/
> > > > > storage/template/
> > > > > > > > > >     >> >>> > HttpTemplateDownloader.java#L93
> > > > > > > > > >     >> >>> > https://goo.gl/dSi0r5
> > > > > > > > > >     >> >>> >  Might need to add
> > > > > > > > > >     >> >>> > client.setRedirectStrategy(new
> > > > > LaxRedirectStrategy());
> > > > > > > > > >     >> >>> >
> > > > > > > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will
> Stevens
> > > > > > > > > >     >> >>> > <ws...@cloudops.com>
> > > > > > > > > >     >> >>> > wrote:
> > > > > > > > > >     >> >>> >
> > > > > > > > > >     >> >>> > > We haven't opened a ticket yet because we
> > > > > > > > > > don't
> > > > > > have a
> > > > > > > > > > strategy
> > > > > > > > > >     >> yet.
> > > > > > > > > >     >> >>> > >
> > > > > > > > > >     >> >>> > > What do you guys think of this:
> > > > > > > > > >     >> >>> > > - We setup a new github repo in the
> 'apache'
> > > > > > > > > > org
> > > > > > which
> > > > > > > > > > consists
> > > > > > > > > >     >> >>> > > of a
> > > > > > > > > >     >> >>> > single
> > > > > > > > > >     >> >>> > > file with a list of active/supported
> > mirrors.
> > > > > > > > > >     >> >>> > > - I write a small web server, distributed
> as
> > > > > > > > > > a
> > > > > > binary,
> > > > > > > > > > which can
> > > > > > > > > >     >> >>> > > be
> > > > > > > > > >     >> >>> > hosted
> > > > > > > > > >     >> >>> > > by ASF Infra.  This web server will query
> > the
> > > > > > current
> > > > > > > > list
> > > > > > > > > > of
> > > > > > > > > >     >> >>> > > mirrors
> > > > > > > > > >     >> >>> and
> > > > > > > > > >     >> >>> > > will select one and then do a 302 redirect
> > to
> > > > > > > > > > that
> > > > > > > > mirror.
> > > > > > > > > >     >> >>> > >
> > > > > > > > > >     >> >>> > > The act of 'choosing' a mirror could be
> done
> > > > > > > > > > in a
> > > > > > > number
> > > > > > > > > of
> > > > > > > > > > ways.
> > > > > > > > > >     >> >>> > > - If we want to define an order, then it
> > > > > > > > > > could
> > > > > just
> > > > > > > try
> > > > > > > > > > from the
> > > > > > > > > >     >> >>> > > top of
> > > > > > > > > >     >> >>> > the
> > > > > > > > > >     >> >>> > > list and work its way down.  It would curl
> > > > > > > > > > the
> > > > > > target
> > > > > > > to
> > > > > > > > > > make
> > > > > > > > > >     >> >>> > > sure it
> > > > > > > > > >     >> >>> > gets
> > > > > > > > > >     >> >>> > > a 200 and if it does, it would do a 302
> > > > redirect.
> > > > > > > > > >     >> >>> > > - Or, if we want to distribute the load
> > > > > > > > > > across the
> > > > > > > > > mirrors,
> > > > > > > > > > we
> > > > > > > > > >     >> >>> > > could
> > > > > > > > > >     >> >>> pick
> > > > > > > > > >     >> >>> > > from the list randomly.  Again, doing a
> curl
> > > > > > > > > > to
> > > > > > verify
> > > > > > > > the
> > > > > > > > > > mirror
> > > > > > > > > >     >> >>> > > is up
> > > > > > > > > >     >> >>> > and
> > > > > > > > > >     >> >>> > > then doing a redirect.
> > > > > > > > > >     >> >>> > > - If we want to get fancy, we could do a
> > > > > > > > > > reverse
> > > > > IP
> > > > > > > > lookup
> > > > > > > > > > and
> > > > > > > > > >     >> >>> > > try to
> > > > > > > > > >     >> >>> > match
> > > > > > > > > >     >> >>> > > the requester with their closest
> > geographical
> > > > > > mirror.
> > > > > > > > > >     >> >>> > >
> > > > > > > > > >     >> >>> > > Thoughts?
> > > > > > > > > >     >> >>> > >
> > > > > > > > > >     >> >>> > > *Will STEVENS*
> > > > > > > > > >     >> >>> > > Lead Developer
> > > > > > > > > >     >> >>> > >
> > > > > > > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
> > > > > > > > > >     >> >>> > >
> > > > > > > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM,
> Chiradeep
> > > > > Vittal <
> > > > > > > > > >     >> >>> chiradeepv@gmail.com
> > > > > > > > > >     >> >>> > >
> > > > > > > > > >     >> >>> > > wrote:
> > > > > > > > > >     >> >>> > >
> > > > > > > > > >     >> >>> > > > What steps are needed to set up a
> mirror?
> > > > > > > > > > What
> > > > > > does
> > > > > > > > > Infra
> > > > > > > > > > need
> > > > > > > > > >     >> to do?
> > > > > > > > > >     >> >>> > Has
> > > > > > > > > >     >> >>> > > > anybody filed a ticket with Infra?
> > > > > > > > > >     >> >>> > > >
> > > > > > > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja
> > > > Pullela <
> > > > > > > > > >     >> >>> > > > raja.pullela@accelerite.com>
> > > > > > > > > >     >> >>> > > > wrote:
> > > > > > > > > >     >> >>> > > >
> > > > > > > > > >     >> >>> > > > > Hi will,
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > > > I believe, we didn’t get to close
> > > > > > > > > > ‘getting a
> > > > > > > mirror
> > > > > > > > on
> > > > > > > > > > Apache’
> > > > > > > > > >     >> >>> > because
> > > > > > > > > >     >> >>> > > we
> > > > > > > > > >     >> >>> > > > > needed someone on the Apache Infra
> side
> > > > > > > > > > to
> > > > > close
> > > > > > > > this.
> > > > > > > > > > BTW,
> > > > > > > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux
> > > > > > > > > > manages
> > > > > > this?)
> > > > > > > > has
> > > > > > > > > >     >> >>> > > > > all/most of
> > > > > > > > > >     >> >>> > the
> > > > > > > > > >     >> >>> > > > > content.  Once we can close on the
> > Apache
> > > > > mirror
> > > > > > > for
> > > > > > > > > > hosting
> > > > > > > > > >     >> >>> > > > > the
> > > > > > > > > >     >> >>> > > > content, I
> > > > > > > > > >     >> >>> > > > > can help assist getting the content
> > there.
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > > > For now, we have replicated the
> > > > > > > download.cloud.com
> > > > > > > > > > content to
> > > > > > > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
> > > > > > > > > >     >> >>> > > > > Also, we are working on a set of
> > > > > steps/procedure
> > > > > > > to
> > > > > > > > > > help with
> > > > > > > > > >     >> >>> > > > > this change.  I will update everyone
> in
> > > > > > > > > > about
> > > > > a
> > > > > > > > week’s
> > > > > > > > > > time
> > > > > > > > > >     >> >>> > > > > on the
> > > > > > > > > >     >> >>> > details.
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > > > Best,
> > > > > > > > > >     >> >>> > > > > Raja Pullela
> > > > > > > > > >     >> >>> > > > > Engineering Team,
> > > > > > > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road,
> Santa
> > > > > > > > > > Clara,
> > > > > > CA,
> > > > > > > > > 95054
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "
> > > > > williamstevens@gmail.com
> > > > > > > on
> > > > > > > > > > behalf of
> > > > > > > > > >     >> >>> > > > > Will Stevens" <
> williamstevens@gmail.com
> > > > > > > > > > on
> > > > > > behalf
> > > > > > > > of
> > > > > > > > > >     >> >>> > wstevens@cloudops.com>
> > > > > > > > > >     >> >>> > > > > wrote:
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > > > unfortunately the template mirror
> > > > > > > > > > conversation
> > > > > > got
> > > > > > > > > > caught up
> > > > > > > > > >     >> >>> > > > > in
> > > > > > > > > >     >> >>> > details
> > > > > > > > > >     >> >>> > > > and
> > > > > > > > > >     >> >>> > > > > nobody took the lead on implementing a
> > > > > solution.
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > > > citrix has been pinging me every
> couple
> > > > > > > > > > months
> > > > > > to
> > > > > > > > say
> > > > > > > > > > 'dude,
> > > > > > > > > >     >> >>> > > > > we
> > > > > > > > > >     >> >>> need
> > > > > > > > > >     >> >>> > to
> > > > > > > > > >     >> >>> > > > > remove the dependency on
> > > > > > > > > > download.citrix.com
> > > > > ',
> > > > > > > but
> > > > > > > > i
> > > > > > > > > > have not
> > > > > > > > > >     >> >>> > > > > had
> > > > > > > > > >     >> >>> > the
> > > > > > > > > >     >> >>> > > > > cycles to get in and solve the
> problem.
> > > > > > > > > > the
> > > > > > > > shutdown
> > > > > > > > > > of that
> > > > > > > > > >     >> >>> > > > > is
> > > > > > > > > >     >> >>> > > imminent
> > > > > > > > > >     >> >>> > > > > right now, so we need to solve it
> asap.
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > > > *Will STEVENS*
> > > > > > > > > >     >> >>> > > > > Lead Developer
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul
> > > > Angus <
> > > > > > > > > >     >> >>> > paul.angus@shapeblue.com
> > > > > > > > > >     >> >>> > > >
> > > > > > > > > >     >> >>> > > > > wrote:
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > > > > Hi Nathan,
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > > Ideally, if you put the template
> > > > > > > > > > location in
> > > > > > (or
> > > > > > > > use
> > > > > > > > > a
> > > > > > > > > >     >> >>> > > > > > template
> > > > > > > > > >     >> >>> > > defined
> > > > > > > > > >     >> >>> > > > > > in)  test_data.py then the actual
> > > > > > > > > > location
> > > > > can
> > > > > > > be
> > > > > > > > > >     >> >>> > > > > > overridden by
> > > > > > > > > >     >> >>> > > anyone
> > > > > > > > > >     >> >>> > > > > > testing.
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > > For Trillian, we've copied all of
> the
> > > > > > templates
> > > > > > > > that
> > > > > > > > > > people
> > > > > > > > > >     >> >>> > > > > > have
> > > > > > > > > >     >> >>> > > define
> > > > > > > > > >     >> >>> > > > > to
> > > > > > > > > >     >> >>> > > > > > a local repo and then replace the
> URLs
> > > > > > > > > > in test_data.py to
> > > > > > > > > >     >> >>> > > > > > reduce
> > > > > > > > > >     >> >>> > > > > bandwidth
> > > > > > > > > >     >> >>> > > > > > use and download times.
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > > Ie:
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > >             "bootableIso":
> > > > > > > > > >     >> >>> > > > > >                 {
> > > > > > > > > >     >> >>> > > > > >                     "displaytext":
> > "Test
> > > > > > > Bootable
> > > > > > > > > > ISO",
> > > > > > > > > >     >> >>> > > > > >                     "name":
> "testISO",
> > > > > > > > > >     >> >>> > > > > >                     "bootable":
> True,
> > > > > > > > > >     >> >>> > > > > >                     "ispublic":
> False,
> > > > > > > > > >     >> >>> > > > > >                     "url": "{{
> > > > > > > > > marvin_images_location
> > > > > > > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
> > > > > > > > > >     >> >>> > > > > >                     "ostype": 'Other
> > > Linux
> > > > > > > > > (64-bit)',
> > > > > > > > > >     >> >>> > > > > >                     "mode":
> > > > 'HTTP_DOWNLOAD'
> > > > > > > > > >     >> >>> > > > > >         },
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > > I thought that we had come up with a
> > > > > solution
> > > > > > > for
> > > > > > > > > >     >> >>> > download.cloud.com
> > > > > > > > > >     >> >>> > > ,
> > > > > > > > > >     >> >>> > > > by
> > > > > > > > > >     >> >>> > > > > > having a mirrorlist hosted in
> > Community
> > > > > Apache
> > > > > > > > > > 'space' with
> > > > > > > > > >     >> >>> anyone
> > > > > > > > > >     >> >>> > > able
> > > > > > > > > >     >> >>> > > > > to
> > > > > > > > > >     >> >>> > > > > > out themselves forward as a mirror.
> > > > > > > > > >     >> >>> > > > > > But I must admit I lost track of
> > > > > > > > > > whether
> > > > > > anyone
> > > > > > > > made
> > > > > > > > > > the
> > > > > > > > > >     >> >>> requisite
> > > > > > > > > >     >> >>> > > > > changes
> > > > > > > > > >     >> >>> > > > > > in code....
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > > Kind regards,
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > > Paul Angus
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > > paul.angus@shapeblue.com
> > > > > > > > > >     >> >>> > > > > > www.shapeblue.com
> > > > > > > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden,
> > London
> > > > > WC2N
> > > > > > > > 4HSUK
> > > > > > > > > >     >> >>> > > > > > @shapeblue
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > > -----Original Message-----
> > > > > > > > > >     >> >>> > > > > > From: williamstevens@gmail.com
> > > > > > > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
> > > > > > > > > >     >> >>> > On
> > > > > > > > > >     >> >>> > > > > > Behalf Of Will Stevens
> > > > > > > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > >     >> >>> > > > > > Subject: Re: Modern template hosting
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > > this is a hard questions.  in
> general,
> > > > > > > > > > we
> > > > > > should
> > > > > > > > be
> > > > > > > > > > setting
> > > > > > > > > >     >> >>> > > > > > up a
> > > > > > > > > >     >> >>> > > mirror
> > > > > > > > > >     >> >>> > > > > on
> > > > > > > > > >     >> >>> > > > > > some cloudstack/apache domain and
> then
> > > > > mirror
> > > > > > to
> > > > > > > > > other
> > > > > > > > > >     >> >>> > > > > > provided
> > > > > > > > > >     >> >>> > > > > templates.
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > > we MUST come up with a solution to
> > > > deprecate
> > > > > > > > > >     >> >>> > > > > > 'download.cloud.com
> > > > > > > > > >     >> >>> ',
> > > > > > > > > >     >> >>> > > > that
> > > > > > > > > >     >> >>> > > > > > is going to be going away any day
> now.
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > > i don't know the right way to solve
> > > > > > > > > > this to
> > > > > be
> > > > > > > > > > honest, but
> > > > > > > > > >     >> >>> > > > > > if you
> > > > > > > > > >     >> >>> > > have
> > > > > > > > > >     >> >>> > > > > > ideas, i am willing to help.
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > > *Will STEVENS*
> > > > > > > > > >     >> >>> > > > > > Lead Developer
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM,
> > > > > > > > > > Nathan
> > > > > > > Johnson <
> > > > > > > > > >     >> >>> njohnson@ena.com
> > > > > > > > > >     >> >>> > >
> > > > > > > > > >     >> >>> > > > > wrote:
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > > > > So not to re-open a can of worms,
> > but
> > > > > > > > > > I’m
> > > > > > in a
> > > > > > > > > > situation
> > > > > > > > > >     >> >>> > > > > > > where
> > > > > > > > > >     >> >>> I
> > > > > > > > > >     >> >>> > > need
> > > > > > > > > >     >> >>> > > > > > > to come up with a Marvin component
> > > > > > > > > > test
> > > > > that
> > > > > > > > > > depends on a
> > > > > > > > > >     >> >>> > template
> > > > > > > > > >     >> >>> > > > > > > based on a kernel that’s
> relatively
> > > > > > > > > > new,
> > > > > > i.e.,
> > > > > > > > > > newer than
> > > > > > > > > >     >> >>> Centos
> > > > > > > > > >     >> >>> > > 5.3
> > > > > > > > > >     >> >>> > > > /
> > > > > > > > > >     >> >>> > > > > > Ubuntu 10.04 .
> > > > > > > > > >     >> >>> > > > > > > I see openvm.eu has a suitable
> > > > > > > > > > template
> > > > > > > (Ubuntu
> > > > > > > > > > 16.0.4
> > > > > > > > > >     >> >>> > > > > > > for
> > > > > > > > > >     >> >>> KVM),
> > > > > > > > > >     >> >>> > > but
> > > > > > > > > >     >> >>> > > > > > > from looking at the thread
> > "Migrating
> > > > > > > CloudStack
> > > > > > > > > > content
> > > > > > > > > >     >> >>> > > > > > > from download.cloud.com” it looks
> > > > > > > > > > like
> > > > > > there
> > > > > > > is
> > > > > > > > > >     >> >>> > > > > > > resistance to using
> > > > > > > > > >     >> >>> > > this
> > > > > > > > > >     >> >>> > > > at
> > > > > > > > > >     >> >>> > > > > > > least for hosting system vm
> > templates
> > > > > > > > > > over
> > > > > > > > > concerns
> > > > > > > > > > of
> > > > > > > > > >     >> >>> > neutrality.
> > > > > > > > > >     >> >>> > > > > > > Would this be suitable for a
> > > > > > > > > > component
> > > > > test?
> > > > > > > If
> > > > > > > > > > not,
> > > > > > > > > >     >> >>> > > > > > > what is a
> > > > > > > > > >     >> >>> > > > > “blessed”
> > > > > > > > > >     >> >>> > > > > > > template location?
> > > > > > > > > >     >> >>> > > > > > >
> > > > > > > > > >     >> >>> > > > > > > Thanks in advance!
> > > > > > > > > >     >> >>> > > > > > >
> > > > > > > > > >     >> >>> > > > > > > Nathan Johnson
> > > > > > > > > >     >> >>> > > > > > > R&D Engineer
> > > > > > > > > >     >> >>> > > > > > >
> > > > > > > > > >     >> >>> > > > > > >
> > > > > > > > > >     >> >>> > > > > > >
> > > > > > > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > Nashville,
> > > > > > > TN
> > > > > > > > > > 37211
> > > > > > > > > >     >> >>> > > > > > > General Office: 615-312-6000
> > > > > > > > > >     >> >>> > > > > > >
> > > > > > > > > >     >> >>> > > > > > > website | blog | support
> > > > > > > > > >     >> >>> > > > > > >
> > > > > > > > > >     >> >>> > > > > > >
> > > > > > > > > >     >> >>> > > > > > >
> > > > > > > > > >     >> >>> > > > > > >
> > > > > > > > > >     >> >>> > > > > > >
> > > > > > > > > >     >> >>> > > > > > >
> > > > > > > > > >     >> >>> > > > > > >
> > > > > > > > > >     >> >>> > > > > >
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > > > DISCLAIMER
> > > > > > > > > >     >> >>> > > > > ==========
> > > > > > > > > >     >> >>> > > > > This e-mail may contain privileged and
> > > > > > > confidential
> > > > > > > > > >     >> >>> > > > > information
> > > > > > > > > >     >> >>> which
> > > > > > > > > >     >> >>> > > is
> > > > > > > > > >     >> >>> > > > > the property of Accelerite, a
> Persistent
> > > > > Systems
> > > > > > > > > > business. It
> > > > > > > > > >     >> >>> > > > > is
> > > > > > > > > >     >> >>> > > intended
> > > > > > > > > >     >> >>> > > > > only for the use of the individual or
> > > > > > > > > > entity
> > > > > to
> > > > > > > > which
> > > > > > > > > > it is
> > > > > > > > > >     >> >>> > addressed.
> > > > > > > > > >     >> >>> > > If
> > > > > > > > > >     >> >>> > > > > you are not the intended recipient,
> you
> > > > > > > > > > are
> > > > > not
> > > > > > > > > > authorized to
> > > > > > > > > >     >> >>> > > > > read,
> > > > > > > > > >     >> >>> > > > retain,
> > > > > > > > > >     >> >>> > > > > copy, print, distribute or use this
> > > message.
> > > > > If
> > > > > > > you
> > > > > > > > > have
> > > > > > > > > >     >> >>> > > > > received
> > > > > > > > > >     >> >>> > this
> > > > > > > > > >     >> >>> > > > > communication in error, please notify
> > the
> > > > > sender
> > > > > > > and
> > > > > > > > > > delete
> > > > > > > > > >     >> >>> > > > > all
> > > > > > > > > >     >> >>> > copies
> > > > > > > > > >     >> >>> > > of
> > > > > > > > > >     >> >>> > > > > this message. Accelerite, a Persistent
> > > > > > > > > > Systems
> > > > > > > > > business
> > > > > > > > > > does
> > > > > > > > > >     >> >>> > > > > not
> > > > > > > > > >     >> >>> > accept
> > > > > > > > > >     >> >>> > > > any
> > > > > > > > > >     >> >>> > > > > liability for virus infected mails.
> > > > > > > > > >     >> >>> > > > >
> > > > > > > > > >     >> >>> > > >
> > > > > > > > > >     >> >>> > >
> > > > > > > > > >     >> >>> >
> > > > > > > > > >     >> >>>
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >abhinandan.prateek@shapeblue.com
> > > > > > > > > >     >> >www.shapeblue.com
> > > > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N
> 4HSUK
> > > > > > > @shapeblue
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >
> > > > > > > > > >     >> >
> > > > > > > > > >     >>
> > > > > > > > > >     >> abhinandan.prateek@shapeblue.com
> > > > > > > > > >     >> www.shapeblue.com
> > > > > > > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N
> 4HSUK
> > > > > > > > > >     >> @shapeblue
> > > > > > > > > >     >>
> > > > > > > > > >     >>
> > > > > > > > > >     >>
> > > > > > > > > >     >>
> > > > > > > > > >     >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > daan.hoogland@shapeblue.com
> > > > > > > > > > www.shapeblue.com
> > > > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > > > > @shapeblue
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Rafael Weingärtner
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Rafael Weingärtner
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Modern template hosting

Posted by Chiradeep Vittal <ch...@gmail.com>.

To me, that is like saying that a customer won't buy a car because he can't
figure out how to roll down his window, so let's build a voice-controlled
window controller. Can be done, sounds great, but what happens when that
voice controller thing in the cloud goes down?


On Fri, Mar 3, 2017 at 3:09 PM, Will Stevens <wi...@gmail.com>
wrote:

> I agree with Paul. Look at the list of things they have to
> learn/master/care about. We don't want to add to that list, we want to
> remove from that list. Make the project/product more accessible. Reduce the
> barrier to entry...
>
> On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:
>
> > Seriously?
> > apt-get install apache2
> >
> > To install CloudStack, they need to know
> > - DB installation, perhaps some SQL
> > - VLAN configuration on their switches
> > - Ins-and-outs between port forwarding, static ips,
> > - NFS
> > - package management
> > - VHDs, qcow2. vmdk
> > - hyperviosrs
> > - and on and on.
> >
> > If they can figure all that out and not figure out a web server, they can
> > always come to the mailing list.
> >
> >
> >
> >
> > On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <pa...@shapeblue.com>
> > wrote:
> >
> > > The issue is not with supporting highly experienced cloud operators. It
> > is
> > > an issue for new users and other 'relatively' inexperienced operators.
> > >
> > > I have helped enough newbies and cloud operators who have been running
> > > their cloud for a while to know that the barriers to entry are too high
> > as
> > > they are. And telling anyone that they need to create a web server so
> > that
> > > they can add their initial template to get started or in order to
> create
> > a
> > > new zone, just isn't going to fly.
> > >
> > > I'm a huge advocate of the 'download.cloudstack.org' endpoint which
> the
> > > community can add/remove mirrors to or from for system VMs or built-in
> > > templates.  Can the same system be used for binary repos ? although
> there
> > > is an added complication of redist vs no-redist there...
> > >
> > >
> > >
> > >
> > >
> > >
> > > paul.angus@shapeblue.com
> > > www.shapeblue.com
> > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > @shapeblue
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Chiradeep Vittal [mailto:chiradeepv@gmail.com]
> > > Sent: 03 March 2017 18:28
> > > To: dev <de...@cloudstack.apache.org>
> > > Subject: Re: Modern template hosting
> > >
> > > I do feel like this is early optimization. Mirrors rarely fail. I'd
> > expect
> > > a single web server hosted on Apache Infra without any monitors to fail
> > > more often than a mirror. We already expect Wido's systemvm repository
> to
> > > be up all the time. And it has been. Similarly, I don't believe Nux's
> > > repository has ever been down. And if Accelerite wants to host on S3,
> > that
> > > one is pretty solid as well.
> > >
> > > This is an infrequent operation in a cloud. After the cloud is
> installed,
> > > the download servers are only needed for a new zone. If we trust the
> user
> > > to run a cloud, surely he/she can run a web server to serve some
> built-in
> > > templates. And if her cloud is successful, she needs to figure out how
> to
> > > host her templates anyway and not rely on 3rd parties.
> > >
> > >
> > > On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <ws...@cloudops.com>
> > > wrote:
> > >
> > > > 1) If the legacy implementations do not support redirects, that does
> > > > cause a problem.  A potential solution in that case is to have the
> web
> > > > server actually proxy the download, but that is not ideal and I would
> > > > like to avoid it if possible.  Thanks for bringing that up Chiradeep.
> > > >
> > > > 2) I think we need to have a single URL which people can target.
> Once
> > > > they make the switch to the new URL, we want the implementation to be
> > > > able to handle mirror failures without affecting the end client.  We
> > > > want to avoid the situation where an ACS user will ever have to
> change
> > > > this URL more than once.  Mirror failures SHOULD NOT affect the ACS
> > > > users assuming there is still at least one mirror who can serve the
> > > requested resource.
> > > >
> > > > These are obviously my personal opinions and others will probably
> have
> > > > differing opinions.
> > > >
> > > > *Will STEVENS*
> > > > Lead Developer
> > > >
> > > > <https://goo.gl/NYZ8KK>
> > > >
> > > > On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal
> > > > <ch...@gmail.com>
> > > > wrote:
> > > >
> > > > > 1. If you are targeting legacy installations, they are not able to
> > > > > follow redirects. The line of code that added this capability was
> > > > > added on
> > > > 11/16.
> > > > > 2. If you trust the users to edit the database to change the URL,
> > > > > you can trust them to change it to anything. Just document a known
> > > > > list of good template locations.
> > > > >
> > > > > On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <
> wstevens@cloudops.com>
> > > > > wrote:
> > > > >
> > > > > > So the main issue I see with this is the following.
> > > > > >
> > > > > > This implementation is designed to target legacy installations
> > > > > > which
> > > > will
> > > > > > be affected when download.cloud.com disappears.  These people
> will
> > > > need
> > > > > to
> > > > > > make a DB change to replace the 'download.cloud.com' with some
> > > > > > other
> > > > url
> > > > > > (maybe 'download.cloudstack.org').  Once the DB has been
> updated,
> > > > > > we
> > > > can
> > > > > > not expect anything else of the client, they need to be able to
> > > > > > just continue operation as they were without any need to rebuild
> > > > > > or upgrade their ACS.
> > > > > >
> > > > > > If we try to force the decision to the client, then we break the
> > > > > > legacy implementations.  Since the SSVM is likely going to be the
> > > > > > client in
> > > > some
> > > > > > cases, and since it does not already have the logic to handle the
> > > > > > 300 approach correctly, I feel like it is not a viable solution
> > > > > > for the
> > > > > legacy
> > > > > > deployments.
> > > > > >
> > > > > > In addition to that.  I don't think the client is in any better
> > > > position
> > > > > to
> > > > > > make the mirror decision than I am.  I am likely able to make a
> > > > > > more 'educated' decision than the client would be able to because
> > > > > > I can do different tests on the endpoint(s) before making a final
> > > decision.
> > > > > >
> > > > > > Does this make sense or am I missing something obvious?
> > > > > >
> > > > > > *Will STEVENS*
> > > > > > Lead Developer
> > > > > >
> > > > > > <https://goo.gl/NYZ8KK>
> > > > > >
> > > > > > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
> > > > > > rafaelweingartner@gmail.com> wrote:
> > > > > >
> > > > > > > I think I understood you, but I did not understand you.
> > > > > > >
> > > > > > > Let me see if I can create an example to illustrate how we
> could
> > > > > > > do
> > > > > using
> > > > > > > the HTTP 300 code.
> > > > > > >
> > > > > > > The HTTP 300 code indicates clients that he/she/it has multiple
> > > > choices
> > > > > > to
> > > > > > > access the requested resource. If the request is a HEAD, this
> is
> > > > > > > the
> > > > > only
> > > > > > > thing that the server returns. If the request if a GET, then
> the
> > > > server
> > > > > > > returns the choices for the client. The response type is
> defined
> > > > > > > on
> > > > the
> > > > > > > server based on the Content-Type, user agent and others.
> > > > > > >
> > > > > > > Let’s say the server receives a GET request and informs the
> > > > > content-type
> > > > > > as
> > > > > > > JSON (application/JSON). Your application could get the mirrors
> > > > > > > list,
> > > > > do
> > > > > > > the appending process for the requested resource path, and then
> > > > return
> > > > > a
> > > > > > > JSON array with the possible mirror the client can use. Then,
> > > > > > > the
> > > > > client
> > > > > > > gets this list and does the processing required to select a
> > mirror.
> > > > > > >
> > > > > > > Now let’s say that a human access the link using a browser.
> > > > > > > Then, the server should receive something like (text/plain) as
> > > > > > > the
> > > > content-type;
> > > > > I
> > > > > > > think depending on the browser this field may differ a little
> > > > > > > (it
> > > > would
> > > > > > > require some checking). Anyways, the server detects that it is
> a
> > > > > “human”
> > > > > > > requesting the resource, then we could serve an HTML page with
> a
> > > > > > Javascript
> > > > > > > that uses the mirror list. This Javascript could do some
> > > > > > > checking and choose the best mirror for that specific.
> > > > > > >
> > > > > > > Why do I think this approach is interesting?
> > > > > > > The application you developed would be used only to retrieve
> > > > > > > valid
> > > > and
> > > > > > > trusted sources of system VM images (a trusted repository for
> > > > > > > mirrors
> > > > > > that
> > > > > > > we as a community vouch for).  Also, we shift the decision
> > > > > > > process regarding mirrors from the server to the client. Then,
> > > > > > > it is up to
> > > > > > clients
> > > > > > > to select mirrors, and not up to us (or some of our
> > > implementations).
> > > > > > >
> > > > > > > Did this help?
> > > > > > > BTW: I am not saying I am against the way you proposed, which
> > > > > > > would
> > > > > work
> > > > > > > fine. It is merely a suggestion using a different perspective.
> > > > > > >
> > > > > > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
> > > > > williamstevens@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > @rafael: in general, I don't think the client should ever be
> > > > making a
> > > > > > > > choice about a mirror. We have to assume we are working with
> a
> > > > > scripted
> > > > > > > > application and anything hard coded in that implementation on
> > > > > > > > the
> > > > > > client
> > > > > > > > side is a risk.
> > > > > > > >
> > > > > > > > I may not be understanding the use of the 300 approach, so I
> > > > > > > > will
> > > > > > > research
> > > > > > > > it to see if I can make it fit.
> > > > > > > >
> > > > > > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
> > > > > > > rafaelweingartner@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > Will great job.
> > > > > > > >
> > > > > > > > I had the same doubt as Daan.
> > > > > > > >
> > > > > > > > Have you considered using HTTP 300 working mode? Then, we
> > > > > > > > could let
> > > > > the
> > > > > > > > client decide which mirror is the best(closest?). Section
> > > “10.3.1”
> > > > of
> > > > > > [1]
> > > > > > > > talks about it; this would put the pressure on deciding upon
> a
> > > > mirror
> > > > > > on
> > > > > > > > the client side, but I think it would be fair.
> > > > > > > >
> > > > > > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
> > > > > > > >
> > > > > > > > PS: You really like very short variables names!
> > > > > > > >
> > > > > > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
> > > > > > > daan.hoogland@shapeblue.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Nice little thing Will,
> > > > > > > > >
> > > > > > > > > One question: if I read the code correctly it ‘go’es and
> > > > > > > > > tries
> > > > all
> > > > > > > > mirrors
> > > > > > > > > at once and whichever responses first is redirected to the
> > > > client.
> > > > > > This
> > > > > > > > > might well be the same every time. This might be the one
> > > > > > > > > closest
> > > > to
> > > > > > the
> > > > > > > > > mirror302 and not to the client. Is that correct or did I
> > > > > > > > > miss a
> > > > > > > weighing
> > > > > > > > > algorithm hidden in there?
> > > > > > > > >
> > > > > > > > > Good coding,
> > > > > > > > >
> > > > > > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org>
> wrote:
> > > > > > > > >
> > > > > > > > >     Hey All,
> > > > > > > > >     Please review this repo and tell me what you think:
> > > > > > > > >     https://github.com/swill/mirror302
> > > > > > > > >
> > > > > > > > >     Let me know if you have questions or you would like me
> > > > > > > > > to
> > > > make
> > > > > > > > changes
> > > > > > > > >     before I bring this topic up with ASF Infra and open a
> > > > > > > > > ticket
> > > > > to
> > > > > > > get
> > > > > > > > > this
> > > > > > > > >     implemented.
> > > > > > > > >
> > > > > > > > >     Cheers,
> > > > > > > > >
> > > > > > > > >     *Will Stevens*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> > > > > > > wstevens@cloudops.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >     > I am building a short term solution right now.  I
> > > > > > > > > hopefully
> > > > > > will
> > > > > > > > > have the
> > > > > > > > >     > building blocks in place by the weekend so we can
> > > > > > > > > start
> > > > > working
> > > > > > > > with
> > > > > > > > > Infra
> > > > > > > > >     > to get it in place.
> > > > > > > > >     >
> > > > > > > > >     > We will have a web server which we can point
> something
> > > like
> > > > > > > > >     > downloads.cloudstack.org (or whatever url) at.  Then
> > > > > > > > > we
> > > > will
> > > > > > > have
> > > > > > > > > an ASF
> > > > > > > > >     > repo which tracks a mirror list and exposes a static
> > > > > > > > > site
> > > > to
> > > > > be
> > > > > > > > able
> > > > > > > > > to
> > > > > > > > >     > query the mirror list.
> > > > > > > > >     >
> > > > > > > > >     > The web server will receive a request and will do a
> > > > > > > > > 302
> > > > > > redirect
> > > > > > > to
> > > > > > > > > the
> > > > > > > > >     > appropriate resource on one of the mirrors.
> > > > > > > > >     >
> > > > > > > > >     > This gets us started.  Legacy environments will have
> > > > > > > > > to do
> > > > a
> > > > > DB
> > > > > > > > > change to
> > > > > > > > >     > change from 'download.cloud.com' to '
> > > > > downloads.cloudstack.org'
> > > > > > > (or
> > > > > > > > >     > whatever), but otherwise it should be pretty simple.
> > > > > > > > >     >
> > > > > > > > >     > We can improve how we deliver templates going
> forward,
> > > > > > > > > but
> > > > > this
> > > > > > > > > obviously
> > > > > > > > >     > requires some discussion still.  I will try to buy us
> > > > > > > > > some
> > > > > time
> > > > > > > > with
> > > > > > > > > an
> > > > > > > > >     > implementation which solves our problems today...
> > > > > > > > >     >
> > > > > > > > >     > *Will STEVENS*
> > > > > > > > >     > Lead Developer
> > > > > > > > >     >
> > > > > > > > >     > <https://goo.gl/NYZ8KK>
> > > > > > > > >     >
> > > > > > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > > > > > > > >     > abhinandan.prateek@shapeblue.com> wrote:
> > > > > > > > >     >
> > > > > > > > >     >>
> > > > > > > > >     >> Initial seeding is a manual step and that is not
> > > > > > > > > going to
> > > > > > change
> > > > > > > > in
> > > > > > > > > near
> > > > > > > > >     >> future. A handy list of official places from where
> > > > > > > > > these
> > > > > > > templates
> > > > > > > > > can be
> > > > > > > > >     >> downloaded won’t harm this part of the installation.
> > > > > > > > > Just
> > > > > > allow
> > > > > > > > > anyone
> > > > > > > > >     >> credible who follows democratic process to publish
> > > > > > > > > their
> > > > > > > templates
> > > > > > > > > with
> > > > > > > > >     >> apache’s blessings.
> > > > > > > > >     >>
> > > > > > > > >     >>
> > > > > > > > >     >> Coming to urls hard coded in db, the popular
> > > > > > > > > suggestion is
> > > > > to
> > > > > > > > point
> > > > > > > > > them
> > > > > > > > >     >> to mirrors. This looks good but this is going to
> take
> > > > > > > > > some
> > > > > > > effort
> > > > > > > > > that has
> > > > > > > > >     >> not materialised yet. Once this change is made it
> > > > > > > > > will be
> > > > > > going
> > > > > > > in
> > > > > > > > > some
> > > > > > > > >     >> future release like 4.11; the problem still remains
> > > > > > > > > for
> > > > the
> > > > > > > > > releases that
> > > > > > > > >     >> are already out there. In case access to
> > > > download.cloud.com
> > > > > > is
> > > > > > > > > dropped
> > > > > > > > >     >> at some point we will have to document the procedure
> > > > > > > > > such
> > > > > that
> > > > > > > > > people know
> > > > > > > > >     >> how to make these stable releases work. That is
> where
> > > > > > > > > the
> > > > > > > > procedure
> > > > > > > > >     >> documented here https://shankerbalan.net/blog/
> > > > > > > > >     >> seed-cloudstack-templates-offline/ should be
> > > > > > > > > officially
> > > > > > adapted
> > > > > > > > > with a
> > > > > > > > >     >> marketplace to pick up templates of choosing. Maybe
> > > > > > > > > we can
> > > > > > have
> > > > > > > a
> > > > > > > > > pre-setup
> > > > > > > > >     >> script to make choices instead of written wiki. Note
> > > > > > > > > that
> > > > > this
> > > > > > > > will
> > > > > > > > > be
> > > > > > > > >     >> anyway needed by the older releases.
> > > > > > > > >     >>
> > > > > > > > >     >>
> > > > > > > > >     >>
> > > > > > > > >     >>
> > > > > > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
> > > > > paul.angus@shapeblue.com>
> > > > > > > > > wrote:
> > > > > > > > >     >>
> > > > > > > > >     >> >-1
> > > > > > > > >     >> >
> > > > > > > > >     >> >We are trying to get MORE people to use CloudStack.
> > > > > > > > > And
> > > > > > > raising
> > > > > > > > > the
> > > > > > > > >     >> barrier to entry does nothing to help that.
> > > > > > > > >     >> >
> > > > > > > > >     >> >
> > > > > > > > >     >> >Kind regards,
> > > > > > > > >     >> >
> > > > > > > > >     >> >Paul Angus
> > > > > > > > >     >> >
> > > > > > > > >     >> >paul.angus@shapeblue.com
> > > > > > > > >     >> >www.shapeblue.com
> > > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > > >     >> >@shapeblue
> > > > > > > > >     >> >
> > > > > > > > >     >> >
> > > > > > > > >     >> >
> > > > > > > > >     >> >
> > > > > > > > >     >> >-----Original Message-----
> > > > > > > > >     >> >From: Abhinandan Prateek [mailto:
> abhinandan.prateek@
> > > > > > > > shapeblue.com]
> > > > > > > > >     >> >Sent: 01 March 2017 09:39
> > > > > > > > >     >> >To: dev@cloudstack.apache.org
> > > > > > > > >     >> >Subject: Re: Modern template hosting
> > > > > > > > >     >> >
> > > > > > > > >     >> >+1
> > > > > > > > >     >> >
> > > > > > > > >     >> >After seeing this discussion running in circles
> > > > > > > > > several
> > > > > > times,
> > > > > > > I
> > > > > > > > > think
> > > > > > > > >     >> we should at least get started with the simplest
> > option.
> > > > > > > > >     >> >
> > > > > > > > >     >> >
> > > > > > > > >     >> >
> > > > > > > > >     >> >
> > > > > > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
> > > > > > > chiradeepv@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >     >> >
> > > > > > > > >     >> >>The alternative foolproof way is to simply not
> > > > > > > > > provide
> > > > the
> > > > > > > > > automatic
> > > > > > > > >     >> >>download.
> > > > > > > > >     >> >>
> > > > > > > > >     >> >>Just document the mirror list URL where the
> > > > > > > > > templates
> > > > can
> > > > > be
> > > > > > > > > found. I
> > > > > > > > >     >> >>don't know why people reject this option.
> > > > > > > > >     >> >>
> > > > > > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
> > > > > > > > templates-offline/
> > > > > > > > >     >> >>
> > > > > > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> > > > > > > > >     >> >><wi...@gmail.com>
> > > > > > > > >     >> >>wrote:
> > > > > > > > >     >> >>
> > > > > > > > >     >> >>> OK. Thanks for the heads up.
> > > > > > > > >     >> >>>
> > > > > > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > > > > > > chiradeepv@gmail.com>
> > > > > > > > >     >> wrote:
> > > > > > > > >     >> >>>
> > > > > > > > >     >> >>> > Sounds workable. The downloader code in the
> > > > > > > > > SSVM
> > > > won't
> > > > > > > > follow
> > > > > > > > >     >> >>> > redirects I think.
> > > > > > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
> > > > > > > > 5511065fc20787619d
> > > > > > > > > 9cd0444
> > > > > > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/
> > > > storage/template/
> > > > > > > > >     >> >>> > HttpTemplateDownloader.java#L93
> > > > > > > > >     >> >>> > https://goo.gl/dSi0r5
> > > > > > > > >     >> >>> >  Might need to add
> > > > > > > > >     >> >>> > client.setRedirectStrategy(new
> > > > LaxRedirectStrategy());
> > > > > > > > >     >> >>> >
> > > > > > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> > > > > > > > >     >> >>> > <ws...@cloudops.com>
> > > > > > > > >     >> >>> > wrote:
> > > > > > > > >     >> >>> >
> > > > > > > > >     >> >>> > > We haven't opened a ticket yet because we
> > > > > > > > > don't
> > > > > have a
> > > > > > > > > strategy
> > > > > > > > >     >> yet.
> > > > > > > > >     >> >>> > >
> > > > > > > > >     >> >>> > > What do you guys think of this:
> > > > > > > > >     >> >>> > > - We setup a new github repo in the 'apache'
> > > > > > > > > org
> > > > > which
> > > > > > > > > consists
> > > > > > > > >     >> >>> > > of a
> > > > > > > > >     >> >>> > single
> > > > > > > > >     >> >>> > > file with a list of active/supported
> mirrors.
> > > > > > > > >     >> >>> > > - I write a small web server, distributed as
> > > > > > > > > a
> > > > > binary,
> > > > > > > > > which can
> > > > > > > > >     >> >>> > > be
> > > > > > > > >     >> >>> > hosted
> > > > > > > > >     >> >>> > > by ASF Infra.  This web server will query
> the
> > > > > current
> > > > > > > list
> > > > > > > > > of
> > > > > > > > >     >> >>> > > mirrors
> > > > > > > > >     >> >>> and
> > > > > > > > >     >> >>> > > will select one and then do a 302 redirect
> to
> > > > > > > > > that
> > > > > > > mirror.
> > > > > > > > >     >> >>> > >
> > > > > > > > >     >> >>> > > The act of 'choosing' a mirror could be done
> > > > > > > > > in a
> > > > > > number
> > > > > > > > of
> > > > > > > > > ways.
> > > > > > > > >     >> >>> > > - If we want to define an order, then it
> > > > > > > > > could
> > > > just
> > > > > > try
> > > > > > > > > from the
> > > > > > > > >     >> >>> > > top of
> > > > > > > > >     >> >>> > the
> > > > > > > > >     >> >>> > > list and work its way down.  It would curl
> > > > > > > > > the
> > > > > target
> > > > > > to
> > > > > > > > > make
> > > > > > > > >     >> >>> > > sure it
> > > > > > > > >     >> >>> > gets
> > > > > > > > >     >> >>> > > a 200 and if it does, it would do a 302
> > > redirect.
> > > > > > > > >     >> >>> > > - Or, if we want to distribute the load
> > > > > > > > > across the
> > > > > > > > mirrors,
> > > > > > > > > we
> > > > > > > > >     >> >>> > > could
> > > > > > > > >     >> >>> pick
> > > > > > > > >     >> >>> > > from the list randomly.  Again, doing a curl
> > > > > > > > > to
> > > > > verify
> > > > > > > the
> > > > > > > > > mirror
> > > > > > > > >     >> >>> > > is up
> > > > > > > > >     >> >>> > and
> > > > > > > > >     >> >>> > > then doing a redirect.
> > > > > > > > >     >> >>> > > - If we want to get fancy, we could do a
> > > > > > > > > reverse
> > > > IP
> > > > > > > lookup
> > > > > > > > > and
> > > > > > > > >     >> >>> > > try to
> > > > > > > > >     >> >>> > match
> > > > > > > > >     >> >>> > > the requester with their closest
> geographical
> > > > > mirror.
> > > > > > > > >     >> >>> > >
> > > > > > > > >     >> >>> > > Thoughts?
> > > > > > > > >     >> >>> > >
> > > > > > > > >     >> >>> > > *Will STEVENS*
> > > > > > > > >     >> >>> > > Lead Developer
> > > > > > > > >     >> >>> > >
> > > > > > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
> > > > > > > > >     >> >>> > >
> > > > > > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep
> > > > Vittal <
> > > > > > > > >     >> >>> chiradeepv@gmail.com
> > > > > > > > >     >> >>> > >
> > > > > > > > >     >> >>> > > wrote:
> > > > > > > > >     >> >>> > >
> > > > > > > > >     >> >>> > > > What steps are needed to set up a mirror?
> > > > > > > > > What
> > > > > does
> > > > > > > > Infra
> > > > > > > > > need
> > > > > > > > >     >> to do?
> > > > > > > > >     >> >>> > Has
> > > > > > > > >     >> >>> > > > anybody filed a ticket with Infra?
> > > > > > > > >     >> >>> > > >
> > > > > > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja
> > > Pullela <
> > > > > > > > >     >> >>> > > > raja.pullela@accelerite.com>
> > > > > > > > >     >> >>> > > > wrote:
> > > > > > > > >     >> >>> > > >
> > > > > > > > >     >> >>> > > > > Hi will,
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > > > I believe, we didn’t get to close
> > > > > > > > > ‘getting a
> > > > > > mirror
> > > > > > > on
> > > > > > > > > Apache’
> > > > > > > > >     >> >>> > because
> > > > > > > > >     >> >>> > > we
> > > > > > > > >     >> >>> > > > > needed someone on the Apache Infra side
> > > > > > > > > to
> > > > close
> > > > > > > this.
> > > > > > > > > BTW,
> > > > > > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux
> > > > > > > > > manages
> > > > > this?)
> > > > > > > has
> > > > > > > > >     >> >>> > > > > all/most of
> > > > > > > > >     >> >>> > the
> > > > > > > > >     >> >>> > > > > content.  Once we can close on the
> Apache
> > > > mirror
> > > > > > for
> > > > > > > > > hosting
> > > > > > > > >     >> >>> > > > > the
> > > > > > > > >     >> >>> > > > content, I
> > > > > > > > >     >> >>> > > > > can help assist getting the content
> there.
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > > > For now, we have replicated the
> > > > > > download.cloud.com
> > > > > > > > > content to
> > > > > > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
> > > > > > > > >     >> >>> > > > > Also, we are working on a set of
> > > > steps/procedure
> > > > > > to
> > > > > > > > > help with
> > > > > > > > >     >> >>> > > > > this change.  I will update everyone in
> > > > > > > > > about
> > > > a
> > > > > > > week’s
> > > > > > > > > time
> > > > > > > > >     >> >>> > > > > on the
> > > > > > > > >     >> >>> > details.
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > > > Best,
> > > > > > > > >     >> >>> > > > > Raja Pullela
> > > > > > > > >     >> >>> > > > > Engineering Team,
> > > > > > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa
> > > > > > > > > Clara,
> > > > > CA,
> > > > > > > > 95054
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "
> > > > williamstevens@gmail.com
> > > > > > on
> > > > > > > > > behalf of
> > > > > > > > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com
> > > > > > > > > on
> > > > > behalf
> > > > > > > of
> > > > > > > > >     >> >>> > wstevens@cloudops.com>
> > > > > > > > >     >> >>> > > > > wrote:
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > > > unfortunately the template mirror
> > > > > > > > > conversation
> > > > > got
> > > > > > > > > caught up
> > > > > > > > >     >> >>> > > > > in
> > > > > > > > >     >> >>> > details
> > > > > > > > >     >> >>> > > > and
> > > > > > > > >     >> >>> > > > > nobody took the lead on implementing a
> > > > solution.
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > > > citrix has been pinging me every couple
> > > > > > > > > months
> > > > > to
> > > > > > > say
> > > > > > > > > 'dude,
> > > > > > > > >     >> >>> > > > > we
> > > > > > > > >     >> >>> need
> > > > > > > > >     >> >>> > to
> > > > > > > > >     >> >>> > > > > remove the dependency on
> > > > > > > > > download.citrix.com
> > > > ',
> > > > > > but
> > > > > > > i
> > > > > > > > > have not
> > > > > > > > >     >> >>> > > > > had
> > > > > > > > >     >> >>> > the
> > > > > > > > >     >> >>> > > > > cycles to get in and solve the problem.
> > > > > > > > > the
> > > > > > > shutdown
> > > > > > > > > of that
> > > > > > > > >     >> >>> > > > > is
> > > > > > > > >     >> >>> > > imminent
> > > > > > > > >     >> >>> > > > > right now, so we need to solve it asap.
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > > > *Will STEVENS*
> > > > > > > > >     >> >>> > > > > Lead Developer
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul
> > > Angus <
> > > > > > > > >     >> >>> > paul.angus@shapeblue.com
> > > > > > > > >     >> >>> > > >
> > > > > > > > >     >> >>> > > > > wrote:
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > > > > Hi Nathan,
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > > Ideally, if you put the template
> > > > > > > > > location in
> > > > > (or
> > > > > > > use
> > > > > > > > a
> > > > > > > > >     >> >>> > > > > > template
> > > > > > > > >     >> >>> > > defined
> > > > > > > > >     >> >>> > > > > > in)  test_data.py then the actual
> > > > > > > > > location
> > > > can
> > > > > > be
> > > > > > > > >     >> >>> > > > > > overridden by
> > > > > > > > >     >> >>> > > anyone
> > > > > > > > >     >> >>> > > > > > testing.
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > > For Trillian, we've copied all of the
> > > > > templates
> > > > > > > that
> > > > > > > > > people
> > > > > > > > >     >> >>> > > > > > have
> > > > > > > > >     >> >>> > > define
> > > > > > > > >     >> >>> > > > > to
> > > > > > > > >     >> >>> > > > > > a local repo and then replace the URLs
> > > > > > > > > in test_data.py to
> > > > > > > > >     >> >>> > > > > > reduce
> > > > > > > > >     >> >>> > > > > bandwidth
> > > > > > > > >     >> >>> > > > > > use and download times.
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > > Ie:
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > >             "bootableIso":
> > > > > > > > >     >> >>> > > > > >                 {
> > > > > > > > >     >> >>> > > > > >                     "displaytext":
> "Test
> > > > > > Bootable
> > > > > > > > > ISO",
> > > > > > > > >     >> >>> > > > > >                     "name": "testISO",
> > > > > > > > >     >> >>> > > > > >                     "bootable": True,
> > > > > > > > >     >> >>> > > > > >                     "ispublic": False,
> > > > > > > > >     >> >>> > > > > >                     "url": "{{
> > > > > > > > marvin_images_location
> > > > > > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
> > > > > > > > >     >> >>> > > > > >                     "ostype": 'Other
> > Linux
> > > > > > > > (64-bit)',
> > > > > > > > >     >> >>> > > > > >                     "mode":
> > > 'HTTP_DOWNLOAD'
> > > > > > > > >     >> >>> > > > > >         },
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > > I thought that we had come up with a
> > > > solution
> > > > > > for
> > > > > > > > >     >> >>> > download.cloud.com
> > > > > > > > >     >> >>> > > ,
> > > > > > > > >     >> >>> > > > by
> > > > > > > > >     >> >>> > > > > > having a mirrorlist hosted in
> Community
> > > > Apache
> > > > > > > > > 'space' with
> > > > > > > > >     >> >>> anyone
> > > > > > > > >     >> >>> > > able
> > > > > > > > >     >> >>> > > > > to
> > > > > > > > >     >> >>> > > > > > out themselves forward as a mirror.
> > > > > > > > >     >> >>> > > > > > But I must admit I lost track of
> > > > > > > > > whether
> > > > > anyone
> > > > > > > made
> > > > > > > > > the
> > > > > > > > >     >> >>> requisite
> > > > > > > > >     >> >>> > > > > changes
> > > > > > > > >     >> >>> > > > > > in code....
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > > Kind regards,
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > > Paul Angus
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > > paul.angus@shapeblue.com
> > > > > > > > >     >> >>> > > > > > www.shapeblue.com
> > > > > > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden,
> London
> > > > WC2N
> > > > > > > 4HSUK
> > > > > > > > >     >> >>> > > > > > @shapeblue
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > > -----Original Message-----
> > > > > > > > >     >> >>> > > > > > From: williamstevens@gmail.com
> > > > > > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
> > > > > > > > >     >> >>> > On
> > > > > > > > >     >> >>> > > > > > Behalf Of Will Stevens
> > > > > > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
> > > > > > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
> > > > > > > > >     >> >>> > > > > > Subject: Re: Modern template hosting
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > > this is a hard questions.  in general,
> > > > > > > > > we
> > > > > should
> > > > > > > be
> > > > > > > > > setting
> > > > > > > > >     >> >>> > > > > > up a
> > > > > > > > >     >> >>> > > mirror
> > > > > > > > >     >> >>> > > > > on
> > > > > > > > >     >> >>> > > > > > some cloudstack/apache domain and then
> > > > mirror
> > > > > to
> > > > > > > > other
> > > > > > > > >     >> >>> > > > > > provided
> > > > > > > > >     >> >>> > > > > templates.
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > > we MUST come up with a solution to
> > > deprecate
> > > > > > > > >     >> >>> > > > > > 'download.cloud.com
> > > > > > > > >     >> >>> ',
> > > > > > > > >     >> >>> > > > that
> > > > > > > > >     >> >>> > > > > > is going to be going away any day now.
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > > i don't know the right way to solve
> > > > > > > > > this to
> > > > be
> > > > > > > > > honest, but
> > > > > > > > >     >> >>> > > > > > if you
> > > > > > > > >     >> >>> > > have
> > > > > > > > >     >> >>> > > > > > ideas, i am willing to help.
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > > *Will STEVENS*
> > > > > > > > >     >> >>> > > > > > Lead Developer
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM,
> > > > > > > > > Nathan
> > > > > > Johnson <
> > > > > > > > >     >> >>> njohnson@ena.com
> > > > > > > > >     >> >>> > >
> > > > > > > > >     >> >>> > > > > wrote:
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > > > > So not to re-open a can of worms,
> but
> > > > > > > > > I’m
> > > > > in a
> > > > > > > > > situation
> > > > > > > > >     >> >>> > > > > > > where
> > > > > > > > >     >> >>> I
> > > > > > > > >     >> >>> > > need
> > > > > > > > >     >> >>> > > > > > > to come up with a Marvin component
> > > > > > > > > test
> > > > that
> > > > > > > > > depends on a
> > > > > > > > >     >> >>> > template
> > > > > > > > >     >> >>> > > > > > > based on a kernel that’s relatively
> > > > > > > > > new,
> > > > > i.e.,
> > > > > > > > > newer than
> > > > > > > > >     >> >>> Centos
> > > > > > > > >     >> >>> > > 5.3
> > > > > > > > >     >> >>> > > > /
> > > > > > > > >     >> >>> > > > > > Ubuntu 10.04 .
> > > > > > > > >     >> >>> > > > > > > I see openvm.eu has a suitable
> > > > > > > > > template
> > > > > > (Ubuntu
> > > > > > > > > 16.0.4
> > > > > > > > >     >> >>> > > > > > > for
> > > > > > > > >     >> >>> KVM),
> > > > > > > > >     >> >>> > > but
> > > > > > > > >     >> >>> > > > > > > from looking at the thread
> "Migrating
> > > > > > CloudStack
> > > > > > > > > content
> > > > > > > > >     >> >>> > > > > > > from download.cloud.com” it looks
> > > > > > > > > like
> > > > > there
> > > > > > is
> > > > > > > > >     >> >>> > > > > > > resistance to using
> > > > > > > > >     >> >>> > > this
> > > > > > > > >     >> >>> > > > at
> > > > > > > > >     >> >>> > > > > > > least for hosting system vm
> templates
> > > > > > > > > over
> > > > > > > > concerns
> > > > > > > > > of
> > > > > > > > >     >> >>> > neutrality.
> > > > > > > > >     >> >>> > > > > > > Would this be suitable for a
> > > > > > > > > component
> > > > test?
> > > > > > If
> > > > > > > > > not,
> > > > > > > > >     >> >>> > > > > > > what is a
> > > > > > > > >     >> >>> > > > > “blessed”
> > > > > > > > >     >> >>> > > > > > > template location?
> > > > > > > > >     >> >>> > > > > > >
> > > > > > > > >     >> >>> > > > > > > Thanks in advance!
> > > > > > > > >     >> >>> > > > > > >
> > > > > > > > >     >> >>> > > > > > > Nathan Johnson
> > > > > > > > >     >> >>> > > > > > > R&D Engineer
> > > > > > > > >     >> >>> > > > > > >
> > > > > > > > >     >> >>> > > > > > >
> > > > > > > > >     >> >>> > > > > > >
> > > > > > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > Nashville,
> > > > > > TN
> > > > > > > > > 37211
> > > > > > > > >     >> >>> > > > > > > General Office: 615-312-6000
> > > > > > > > >     >> >>> > > > > > >
> > > > > > > > >     >> >>> > > > > > > website | blog | support
> > > > > > > > >     >> >>> > > > > > >
> > > > > > > > >     >> >>> > > > > > >
> > > > > > > > >     >> >>> > > > > > >
> > > > > > > > >     >> >>> > > > > > >
> > > > > > > > >     >> >>> > > > > > >
> > > > > > > > >     >> >>> > > > > > >
> > > > > > > > >     >> >>> > > > > > >
> > > > > > > > >     >> >>> > > > > >
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > > > DISCLAIMER
> > > > > > > > >     >> >>> > > > > ==========
> > > > > > > > >     >> >>> > > > > This e-mail may contain privileged and
> > > > > > confidential
> > > > > > > > >     >> >>> > > > > information
> > > > > > > > >     >> >>> which
> > > > > > > > >     >> >>> > > is
> > > > > > > > >     >> >>> > > > > the property of Accelerite, a Persistent
> > > > Systems
> > > > > > > > > business. It
> > > > > > > > >     >> >>> > > > > is
> > > > > > > > >     >> >>> > > intended
> > > > > > > > >     >> >>> > > > > only for the use of the individual or
> > > > > > > > > entity
> > > > to
> > > > > > > which
> > > > > > > > > it is
> > > > > > > > >     >> >>> > addressed.
> > > > > > > > >     >> >>> > > If
> > > > > > > > >     >> >>> > > > > you are not the intended recipient, you
> > > > > > > > > are
> > > > not
> > > > > > > > > authorized to
> > > > > > > > >     >> >>> > > > > read,
> > > > > > > > >     >> >>> > > > retain,
> > > > > > > > >     >> >>> > > > > copy, print, distribute or use this
> > message.
> > > > If
> > > > > > you
> > > > > > > > have
> > > > > > > > >     >> >>> > > > > received
> > > > > > > > >     >> >>> > this
> > > > > > > > >     >> >>> > > > > communication in error, please notify
> the
> > > > sender
> > > > > > and
> > > > > > > > > delete
> > > > > > > > >     >> >>> > > > > all
> > > > > > > > >     >> >>> > copies
> > > > > > > > >     >> >>> > > of
> > > > > > > > >     >> >>> > > > > this message. Accelerite, a Persistent
> > > > > > > > > Systems
> > > > > > > > business
> > > > > > > > > does
> > > > > > > > >     >> >>> > > > > not
> > > > > > > > >     >> >>> > accept
> > > > > > > > >     >> >>> > > > any
> > > > > > > > >     >> >>> > > > > liability for virus infected mails.
> > > > > > > > >     >> >>> > > > >
> > > > > > > > >     >> >>> > > >
> > > > > > > > >     >> >>> > >
> > > > > > > > >     >> >>> >
> > > > > > > > >     >> >>>
> > > > > > > > >     >> >
> > > > > > > > >     >> >abhinandan.prateek@shapeblue.com
> > > > > > > > >     >> >www.shapeblue.com
> > > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > @shapeblue
> > > > > > > > >     >> >
> > > > > > > > >     >> >
> > > > > > > > >     >> >
> > > > > > > > >     >>
> > > > > > > > >     >> abhinandan.prateek@shapeblue.com
> > > > > > > > >     >> www.shapeblue.com
> > > > > > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > > >     >> @shapeblue
> > > > > > > > >     >>
> > > > > > > > >     >>
> > > > > > > > >     >>
> > > > > > > > >     >>
> > > > > > > > >     >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > daan.hoogland@shapeblue.com
> > > > > > > > > www.shapeblue.com
> > > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > > > @shapeblue
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Rafael Weingärtner
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Rafael Weingärtner
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Modern template hosting

Posted by Daan Hoogland <da...@shapeblue.com>.

I did some PR archeology and discovered #1582. It is a blocker and superseded by this discussion. Do we still think it is valid? And will people that contributed for this send me pull-requests on that PR? It is dated june 7thbut I will update it if needed!

Anybody who thinks this can be moved to future releases?

On 04/03/17 01:33, "Will Stevens" <wi...@gmail.com> wrote:

    I will be guessing and we will see what I can come up with. :)
    
    On Mar 3, 2017 7:09 PM, "Erik Weber" <te...@gmail.com> wrote:
    
    > Feel free to do it, I wouldn't know where to begin to ask such a
    > question tbh :-)
    >
    > --
    > Erik
    >
    > On Sat, Mar 4, 2017 at 12:51 AM, Will Stevens <wi...@gmail.com>
    > wrote:
    > > Erik, yes I think this is the right approach and covers the main problem
    > > case. I can start the conversation with infra unless you would like to.
    > Let
    > > me know.
    > >
    > > We did get some statistics from the other mirrors last year, but I don't
    > > think we ever had visibility into the cloud.com repository traffic.
    > >
    > > On Mar 3, 2017 6:42 PM, "Erik Weber" <te...@gmail.com> wrote:
    > >
    > > It'll most likely only be an issue in the case where a mirror goes
    > > down without any immediate chance of getting back up, right?
    > > Could we check with ASF Infra beforehand if such case is OK to warrant
    > > an urgent Jira ticket to resolve - should it happen?
    > >
    > > We can document all the mirrors and provide the howtos for manually
    > > downloading and seeding the image as a backup.
    > >
    > > Do we have any traffic statistics for the current solution, or is that
    > > not available at S3?
    > >
    > > --
    > > Erik
    > >
    > > On Sat, Mar 4, 2017 at 12:28 AM, Will Stevens <wi...@gmail.com>
    > > wrote:
    > >> Yes Erik, I agree with you. The only thing that could add complexity in
    > >> this is the fact that we don't control the domain, the ASF does (from
    > what
    > >> I understand).  Do we expect this to be managed by ASF infra or would
    > > there
    > >> be another way? Ideally the domain is community controlled so we can
    > adapt
    > >> in case of change without long delays.
    > >>
    > >> I am 100% in agreement that this is the shortest path and likely to be
    > > good
    > >> enough.
    > >>
    > >> On Mar 3, 2017 6:19 PM, "Erik Weber" <te...@gmail.com> wrote:
    > >>
    > >> IMHO; implement it as simple as possible, use DNS RR and assume/expect
    > >> mirror hosts to use the same path.
    > >>
    > >>
    > >> Yes I know; DNS RR isn't bulletproof, but as already mentioned in this
    > >> thread - mirrors rarely go down and they aren't used all that often
    > >> (primarily seeding).
    > >> We can keep a low TTL on the record so that we're able to remove a
    > >> mirror that goes down for a significant period of time (more than
    > >> hours).
    > >>
    > >> And yes I know, some operators might find it a burden having to set up
    > >> a new vhost with another document root, but tbh if they don't care
    > >> enough to do it we'll manage without that mirror.
    > >>
    > >> --
    > >> Erik
    > >>
    > >> On Sat, Mar 4, 2017 at 12:09 AM, Will Stevens <williamstevens@gmail.com
    > >
    > >> wrote:
    > >>> I agree with Paul. Look at the list of things they have to
    > >>> learn/master/care about. We don't want to add to that list, we want to
    > >>> remove from that list. Make the project/product more accessible. Reduce
    > >> the
    > >>> barrier to entry...
    > >>>
    > >>> On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <ch...@gmail.com>
    > wrote:
    > >>>
    > >>>> Seriously?
    > >>>> apt-get install apache2
    > >>>>
    > >>>> To install CloudStack, they need to know
    > >>>> - DB installation, perhaps some SQL
    > >>>> - VLAN configuration on their switches
    > >>>> - Ins-and-outs between port forwarding, static ips,
    > >>>> - NFS
    > >>>> - package management
    > >>>> - VHDs, qcow2. vmdk
    > >>>> - hyperviosrs
    > >>>> - and on and on.
    > >>>>
    > >>>> If they can figure all that out and not figure out a web server, they
    > > can
    > >>>> always come to the mailing list.
    > >>>>
    > >>>>
    > >>>>
    > >>>>
    > >>>> On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <pa...@shapeblue.com>
    > >>>> wrote:
    > >>>>
    > >>>> > The issue is not with supporting highly experienced cloud operators.
    > > It
    > >>>> is
    > >>>> > an issue for new users and other 'relatively' inexperienced
    > operators.
    > >>>> >
    > >>>> > I have helped enough newbies and cloud operators who have been
    > running
    > >>>> > their cloud for a while to know that the barriers to entry are too
    > > high
    > >>>> as
    > >>>> > they are. And telling anyone that they need to create a web server
    > so
    > >>>> that
    > >>>> > they can add their initial template to get started or in order to
    > >> create
    > >>>> a
    > >>>> > new zone, just isn't going to fly.
    > >>>> >
    > >>>> > I'm a huge advocate of the 'download.cloudstack.org' endpoint which
    > > the
    > >>>> > community can add/remove mirrors to or from for system VMs or
    > built-in
    > >>>> > templates.  Can the same system be used for binary repos ? although
    > >> there
    > >>>> > is an added complication of redist vs no-redist there...
    > >>>> >
    > >>>> >
    > >>>> >
    > >>>> >
    > >>>> >
    > >>>> >
    > >>>> > paul.angus@shapeblue.com
    > >>>> > www.shapeblue.com
    > >>>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
    > >>>> > @shapeblue
    > >>>> >
    > >>>> >
    > >>>> >
    > >>>> >
    > >>>> > -----Original Message-----
    > >>>> > From: Chiradeep Vittal [mailto:chiradeepv@gmail.com]
    > >>>> > Sent: 03 March 2017 18:28
    > >>>> > To: dev <de...@cloudstack.apache.org>
    > >>>> > Subject: Re: Modern template hosting
    > >>>> >
    > >>>> > I do feel like this is early optimization. Mirrors rarely fail. I'd
    > >>>> expect
    > >>>> > a single web server hosted on Apache Infra without any monitors to
    > > fail
    > >>>> > more often than a mirror. We already expect Wido's systemvm
    > repository
    > >> to
    > >>>> > be up all the time. And it has been. Similarly, I don't believe
    > Nux's
    > >>>> > repository has ever been down. And if Accelerite wants to host on
    > S3,
    > >>>> that
    > >>>> > one is pretty solid as well.
    > >>>> >
    > >>>> > This is an infrequent operation in a cloud. After the cloud is
    > >> installed,
    > >>>> > the download servers are only needed for a new zone. If we trust the
    > >> user
    > >>>> > to run a cloud, surely he/she can run a web server to serve some
    > >> built-in
    > >>>> > templates. And if her cloud is successful, she needs to figure out
    > how
    > >> to
    > >>>> > host her templates anyway and not rely on 3rd parties.
    > >>>> >
    > >>>> >
    > >>>> > On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <wstevens@cloudops.com
    > >
    > >>>> > wrote:
    > >>>> >
    > >>>> > > 1) If the legacy implementations do not support redirects, that
    > does
    > >>>> > > cause a problem.  A potential solution in that case is to have the
    > >> web
    > >>>> > > server actually proxy the download, but that is not ideal and I
    > > would
    > >>>> > > like to avoid it if possible.  Thanks for bringing that up
    > > Chiradeep.
    > >>>> > >
    > >>>> > > 2) I think we need to have a single URL which people can target.
    > >> Once
    > >>>> > > they make the switch to the new URL, we want the implementation to
    > > be
    > >>>> > > able to handle mirror failures without affecting the end client.
    > We
    > >>>> > > want to avoid the situation where an ACS user will ever have to
    > >> change
    > >>>> > > this URL more than once.  Mirror failures SHOULD NOT affect the
    > ACS
    > >>>> > > users assuming there is still at least one mirror who can serve
    > the
    > >>>> > requested resource.
    > >>>> > >
    > >>>> > > These are obviously my personal opinions and others will probably
    > >> have
    > >>>> > > differing opinions.
    > >>>> > >
    > >>>> > > *Will STEVENS*
    > >>>> > > Lead Developer
    > >>>> > >
    > >>>> > > <https://goo.gl/NYZ8KK>
    > >>>> > >
    > >>>> > > On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal
    > >>>> > > <ch...@gmail.com>
    > >>>> > > wrote:
    > >>>> > >
    > >>>> > > > 1. If you are targeting legacy installations, they are not able
    > to
    > >>>> > > > follow redirects. The line of code that added this capability
    > was
    > >>>> > > > added on
    > >>>> > > 11/16.
    > >>>> > > > 2. If you trust the users to edit the database to change the
    > URL,
    > >>>> > > > you can trust them to change it to anything. Just document a
    > known
    > >>>> > > > list of good template locations.
    > >>>> > > >
    > >>>> > > > On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <
    > > wstevens@cloudops.com
    > >>>
    > >>>> > > > wrote:
    > >>>> > > >
    > >>>> > > > > So the main issue I see with this is the following.
    > >>>> > > > >
    > >>>> > > > > This implementation is designed to target legacy installations
    > >>>> > > > > which
    > >>>> > > will
    > >>>> > > > > be affected when download.cloud.com disappears.  These people
    > >> will
    > >>>> > > need
    > >>>> > > > to
    > >>>> > > > > make a DB change to replace the 'download.cloud.com' with
    > some
    > >>>> > > > > other
    > >>>> > > url
    > >>>> > > > > (maybe 'download.cloudstack.org').  Once the DB has been
    > > updated,
    > >>>> > > > > we
    > >>>> > > can
    > >>>> > > > > not expect anything else of the client, they need to be able
    > to
    > >>>> > > > > just continue operation as they were without any need to
    > rebuild
    > >>>> > > > > or upgrade their ACS.
    > >>>> > > > >
    > >>>> > > > > If we try to force the decision to the client, then we break
    > the
    > >>>> > > > > legacy implementations.  Since the SSVM is likely going to be
    > > the
    > >>>> > > > > client in
    > >>>> > > some
    > >>>> > > > > cases, and since it does not already have the logic to handle
    > > the
    > >>>> > > > > 300 approach correctly, I feel like it is not a viable
    > solution
    > >>>> > > > > for the
    > >>>> > > > legacy
    > >>>> > > > > deployments.
    > >>>> > > > >
    > >>>> > > > > In addition to that.  I don't think the client is in any
    > better
    > >>>> > > position
    > >>>> > > > to
    > >>>> > > > > make the mirror decision than I am.  I am likely able to make
    > a
    > >>>> > > > > more 'educated' decision than the client would be able to
    > > because
    > >>>> > > > > I can do different tests on the endpoint(s) before making a
    > > final
    > >>>> > decision.
    > >>>> > > > >
    > >>>> > > > > Does this make sense or am I missing something obvious?
    > >>>> > > > >
    > >>>> > > > > *Will STEVENS*
    > >>>> > > > > Lead Developer
    > >>>> > > > >
    > >>>> > > > > <https://goo.gl/NYZ8KK>
    > >>>> > > > >
    > >>>> > > > > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
    > >>>> > > > > rafaelweingartner@gmail.com> wrote:
    > >>>> > > > >
    > >>>> > > > > > I think I understood you, but I did not understand you.
    > >>>> > > > > >
    > >>>> > > > > > Let me see if I can create an example to illustrate how we
    > >> could
    > >>>> > > > > > do
    > >>>> > > > using
    > >>>> > > > > > the HTTP 300 code.
    > >>>> > > > > >
    > >>>> > > > > > The HTTP 300 code indicates clients that he/she/it has
    > > multiple
    > >>>> > > choices
    > >>>> > > > > to
    > >>>> > > > > > access the requested resource. If the request is a HEAD,
    > this
    > >> is
    > >>>> > > > > > the
    > >>>> > > > only
    > >>>> > > > > > thing that the server returns. If the request if a GET, then
    > >> the
    > >>>> > > server
    > >>>> > > > > > returns the choices for the client. The response type is
    > >> defined
    > >>>> > > > > > on
    > >>>> > > the
    > >>>> > > > > > server based on the Content-Type, user agent and others.
    > >>>> > > > > >
    > >>>> > > > > > Let’s say the server receives a GET request and informs the
    > >>>> > > > content-type
    > >>>> > > > > as
    > >>>> > > > > > JSON (application/JSON). Your application could get the
    > > mirrors
    > >>>> > > > > > list,
    > >>>> > > > do
    > >>>> > > > > > the appending process for the requested resource path, and
    > > then
    > >>>> > > return
    > >>>> > > > a
    > >>>> > > > > > JSON array with the possible mirror the client can use.
    > Then,
    > >>>> > > > > > the
    > >>>> > > > client
    > >>>> > > > > > gets this list and does the processing required to select a
    > >>>> mirror.
    > >>>> > > > > >
    > >>>> > > > > > Now let’s say that a human access the link using a browser.
    > >>>> > > > > > Then, the server should receive something like (text/plain)
    > as
    > >>>> > > > > > the
    > >>>> > > content-type;
    > >>>> > > > I
    > >>>> > > > > > think depending on the browser this field may differ a
    > little
    > >>>> > > > > > (it
    > >>>> > > would
    > >>>> > > > > > require some checking). Anyways, the server detects that it
    > is
    > >> a
    > >>>> > > > “human”
    > >>>> > > > > > requesting the resource, then we could serve an HTML page
    > with
    > >> a
    > >>>> > > > > Javascript
    > >>>> > > > > > that uses the mirror list. This Javascript could do some
    > >>>> > > > > > checking and choose the best mirror for that specific.
    > >>>> > > > > >
    > >>>> > > > > > Why do I think this approach is interesting?
    > >>>> > > > > > The application you developed would be used only to retrieve
    > >>>> > > > > > valid
    > >>>> > > and
    > >>>> > > > > > trusted sources of system VM images (a trusted repository
    > for
    > >>>> > > > > > mirrors
    > >>>> > > > > that
    > >>>> > > > > > we as a community vouch for).  Also, we shift the decision
    > >>>> > > > > > process regarding mirrors from the server to the client.
    > Then,
    > >>>> > > > > > it is up to
    > >>>> > > > > clients
    > >>>> > > > > > to select mirrors, and not up to us (or some of our
    > >>>> > implementations).
    > >>>> > > > > >
    > >>>> > > > > > Did this help?
    > >>>> > > > > > BTW: I am not saying I am against the way you proposed,
    > which
    > >>>> > > > > > would
    > >>>> > > > work
    > >>>> > > > > > fine. It is merely a suggestion using a different
    > perspective.
    > >>>> > > > > >
    > >>>> > > > > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
    > >>>> > > > williamstevens@gmail.com>
    > >>>> > > > > > wrote:
    > >>>> > > > > >
    > >>>> > > > > > > @rafael: in general, I don't think the client should ever
    > be
    > >>>> > > making a
    > >>>> > > > > > > choice about a mirror. We have to assume we are working
    > with
    > >> a
    > >>>> > > > scripted
    > >>>> > > > > > > application and anything hard coded in that implementation
    > > on
    > >>>> > > > > > > the
    > >>>> > > > > client
    > >>>> > > > > > > side is a risk.
    > >>>> > > > > > >
    > >>>> > > > > > > I may not be understanding the use of the 300 approach,
    > so I
    > >>>> > > > > > > will
    > >>>> > > > > > research
    > >>>> > > > > > > it to see if I can make it fit.
    > >>>> > > > > > >
    > >>>> > > > > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
    > >>>> > > > > > rafaelweingartner@gmail.com>
    > >>>> > > > > > > wrote:
    > >>>> > > > > > >
    > >>>> > > > > > > Will great job.
    > >>>> > > > > > >
    > >>>> > > > > > > I had the same doubt as Daan.
    > >>>> > > > > > >
    > >>>> > > > > > > Have you considered using HTTP 300 working mode? Then, we
    > >>>> > > > > > > could let
    > >>>> > > > the
    > >>>> > > > > > > client decide which mirror is the best(closest?). Section
    > >>>> > “10.3.1”
    > >>>> > > of
    > >>>> > > > > [1]
    > >>>> > > > > > > talks about it; this would put the pressure on deciding
    > upon
    > >> a
    > >>>> > > mirror
    > >>>> > > > > on
    > >>>> > > > > > > the client side, but I think it would be fair.
    > >>>> > > > > > >
    > >>>> > > > > > > [1] https://www.w3.org/Protocols/
    > rfc2616/rfc2616-sec10.html
    > >>>> > > > > > >
    > >>>> > > > > > > PS: You really like very short variables names!
    > >>>> > > > > > >
    > >>>> > > > > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
    > >>>> > > > > > daan.hoogland@shapeblue.com
    > >>>> > > > > > > >
    > >>>> > > > > > > wrote:
    > >>>> > > > > > >
    > >>>> > > > > > > > Nice little thing Will,
    > >>>> > > > > > > >
    > >>>> > > > > > > > One question: if I read the code correctly it ‘go’es and
    > >>>> > > > > > > > tries
    > >>>> > > all
    > >>>> > > > > > > mirrors
    > >>>> > > > > > > > at once and whichever responses first is redirected to
    > the
    > >>>> > > client.
    > >>>> > > > > This
    > >>>> > > > > > > > might well be the same every time. This might be the one
    > >>>> > > > > > > > closest
    > >>>> > > to
    > >>>> > > > > the
    > >>>> > > > > > > > mirror302 and not to the client. Is that correct or did
    > I
    > >>>> > > > > > > > miss a
    > >>>> > > > > > weighing
    > >>>> > > > > > > > algorithm hidden in there?
    > >>>> > > > > > > >
    > >>>> > > > > > > > Good coding,
    > >>>> > > > > > > >
    > >>>> > > > > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org>
    > > wrote:
    > >>>> > > > > > > >
    > >>>> > > > > > > >     Hey All,
    > >>>> > > > > > > >     Please review this repo and tell me what you think:
    > >>>> > > > > > > >     https://github.com/swill/mirror302
    > >>>> > > > > > > >
    > >>>> > > > > > > >     Let me know if you have questions or you would like
    > me
    > >>>> > > > > > > > to
    > >>>> > > make
    > >>>> > > > > > > changes
    > >>>> > > > > > > >     before I bring this topic up with ASF Infra and
    > open a
    > >>>> > > > > > > > ticket
    > >>>> > > > to
    > >>>> > > > > > get
    > >>>> > > > > > > > this
    > >>>> > > > > > > >     implemented.
    > >>>> > > > > > > >
    > >>>> > > > > > > >     Cheers,
    > >>>> > > > > > > >
    > >>>> > > > > > > >     *Will Stevens*
    > >>>> > > > > > > >
    > >>>> > > > > > > >
    > >>>> > > > > > > >
    > >>>> > > > > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
    > >>>> > > > > > wstevens@cloudops.com
    > >>>> > > > > > > >
    > >>>> > > > > > > > wrote:
    > >>>> > > > > > > >
    > >>>> > > > > > > >     > I am building a short term solution right now.  I
    > >>>> > > > > > > > hopefully
    > >>>> > > > > will
    > >>>> > > > > > > > have the
    > >>>> > > > > > > >     > building blocks in place by the weekend so we can
    > >>>> > > > > > > > start
    > >>>> > > > working
    > >>>> > > > > > > with
    > >>>> > > > > > > > Infra
    > >>>> > > > > > > >     > to get it in place.
    > >>>> > > > > > > >     >
    > >>>> > > > > > > >     > We will have a web server which we can point
    > >> something
    > >>>> > like
    > >>>> > > > > > > >     > downloads.cloudstack.org (or whatever url) at.
    > Then
    > >>>> > > > > > > > we
    > >>>> > > will
    > >>>> > > > > > have
    > >>>> > > > > > > > an ASF
    > >>>> > > > > > > >     > repo which tracks a mirror list and exposes a
    > static
    > >>>> > > > > > > > site
    > >>>> > > to
    > >>>> > > > be
    > >>>> > > > > > > able
    > >>>> > > > > > > > to
    > >>>> > > > > > > >     > query the mirror list.
    > >>>> > > > > > > >     >
    > >>>> > > > > > > >     > The web server will receive a request and will do
    > a
    > >>>> > > > > > > > 302
    > >>>> > > > > redirect
    > >>>> > > > > > to
    > >>>> > > > > > > > the
    > >>>> > > > > > > >     > appropriate resource on one of the mirrors.
    > >>>> > > > > > > >     >
    > >>>> > > > > > > >     > This gets us started.  Legacy environments will
    > have
    > >>>> > > > > > > > to do
    > >>>> > > a
    > >>>> > > > DB
    > >>>> > > > > > > > change to
    > >>>> > > > > > > >     > change from 'download.cloud.com' to '
    > >>>> > > > downloads.cloudstack.org'
    > >>>> > > > > > (or
    > >>>> > > > > > > >     > whatever), but otherwise it should be pretty
    > simple.
    > >>>> > > > > > > >     >
    > >>>> > > > > > > >     > We can improve how we deliver templates going
    > >> forward,
    > >>>> > > > > > > > but
    > >>>> > > > this
    > >>>> > > > > > > > obviously
    > >>>> > > > > > > >     > requires some discussion still.  I will try to buy
    > > us
    > >>>> > > > > > > > some
    > >>>> > > > time
    > >>>> > > > > > > with
    > >>>> > > > > > > > an
    > >>>> > > > > > > >     > implementation which solves our problems today...
    > >>>> > > > > > > >     >
    > >>>> > > > > > > >     > *Will STEVENS*
    > >>>> > > > > > > >     > Lead Developer
    > >>>> > > > > > > >     >
    > >>>> > > > > > > >     > <https://goo.gl/NYZ8KK>
    > >>>> > > > > > > >     >
    > >>>> > > > > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan
    > Prateek <
    > >>>> > > > > > > >     > abhinandan.prateek@shapeblue.com> wrote:
    > >>>> > > > > > > >     >
    > >>>> > > > > > > >     >>
    > >>>> > > > > > > >     >> Initial seeding is a manual step and that is not
    > >>>> > > > > > > > going to
    > >>>> > > > > change
    > >>>> > > > > > > in
    > >>>> > > > > > > > near
    > >>>> > > > > > > >     >> future. A handy list of official places from
    > where
    > >>>> > > > > > > > these
    > >>>> > > > > > templates
    > >>>> > > > > > > > can be
    > >>>> > > > > > > >     >> downloaded won’t harm this part of the
    > > installation.
    > >>>> > > > > > > > Just
    > >>>> > > > > allow
    > >>>> > > > > > > > anyone
    > >>>> > > > > > > >     >> credible who follows democratic process to
    > publish
    > >>>> > > > > > > > their
    > >>>> > > > > > templates
    > >>>> > > > > > > > with
    > >>>> > > > > > > >     >> apache’s blessings.
    > >>>> > > > > > > >     >>
    > >>>> > > > > > > >     >>
    > >>>> > > > > > > >     >> Coming to urls hard coded in db, the popular
    > >>>> > > > > > > > suggestion is
    > >>>> > > > to
    > >>>> > > > > > > point
    > >>>> > > > > > > > them
    > >>>> > > > > > > >     >> to mirrors. This looks good but this is going to
    > >> take
    > >>>> > > > > > > > some
    > >>>> > > > > > effort
    > >>>> > > > > > > > that has
    > >>>> > > > > > > >     >> not materialised yet. Once this change is made it
    > >>>> > > > > > > > will be
    > >>>> > > > > going
    > >>>> > > > > > in
    > >>>> > > > > > > > some
    > >>>> > > > > > > >     >> future release like 4.11; the problem still
    > remains
    > >>>> > > > > > > > for
    > >>>> > > the
    > >>>> > > > > > > > releases that
    > >>>> > > > > > > >     >> are already out there. In case access to
    > >>>> > > download.cloud.com
    > >>>> > > > > is
    > >>>> > > > > > > > dropped
    > >>>> > > > > > > >     >> at some point we will have to document the
    > > procedure
    > >>>> > > > > > > > such
    > >>>> > > > that
    > >>>> > > > > > > > people know
    > >>>> > > > > > > >     >> how to make these stable releases work. That is
    > >> where
    > >>>> > > > > > > > the
    > >>>> > > > > > > procedure
    > >>>> > > > > > > >     >> documented here https://shankerbalan.net/blog/
    > >>>> > > > > > > >     >> seed-cloudstack-templates-offline/ should be
    > >>>> > > > > > > > officially
    > >>>> > > > > adapted
    > >>>> > > > > > > > with a
    > >>>> > > > > > > >     >> marketplace to pick up templates of choosing.
    > Maybe
    > >>>> > > > > > > > we can
    > >>>> > > > > have
    > >>>> > > > > > a
    > >>>> > > > > > > > pre-setup
    > >>>> > > > > > > >     >> script to make choices instead of written wiki.
    > > Note
    > >>>> > > > > > > > that
    > >>>> > > > this
    > >>>> > > > > > > will
    > >>>> > > > > > > > be
    > >>>> > > > > > > >     >> anyway needed by the older releases.
    > >>>> > > > > > > >     >>
    > >>>> > > > > > > >     >>
    > >>>> > > > > > > >     >>
    > >>>> > > > > > > >     >>
    > >>>> > > > > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
    > >>>> > > > paul.angus@shapeblue.com>
    > >>>> > > > > > > > wrote:
    > >>>> > > > > > > >     >>
    > >>>> > > > > > > >     >> >-1
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >We are trying to get MORE people to use
    > > CloudStack.
    > >>>> > > > > > > > And
    > >>>> > > > > > raising
    > >>>> > > > > > > > the
    > >>>> > > > > > > >     >> barrier to entry does nothing to help that.
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >Kind regards,
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >Paul Angus
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >paul.angus@shapeblue.com
    > >>>> > > > > > > >     >> >www.shapeblue.com
    > >>>> > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N
    > > 4HSUK
    > >>>> > > > > > > >     >> >@shapeblue
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >-----Original Message-----
    > >>>> > > > > > > >     >> >From: Abhinandan Prateek [mailto:
    > >> abhinandan.prateek@
    > >>>> > > > > > > shapeblue.com]
    > >>>> > > > > > > >     >> >Sent: 01 March 2017 09:39
    > >>>> > > > > > > >     >> >To: dev@cloudstack.apache.org
    > >>>> > > > > > > >     >> >Subject: Re: Modern template hosting
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >+1
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >After seeing this discussion running in circles
    > >>>> > > > > > > > several
    > >>>> > > > > times,
    > >>>> > > > > > I
    > >>>> > > > > > > > think
    > >>>> > > > > > > >     >> we should at least get started with the simplest
    > >>>> option.
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
    > >>>> > > > > > chiradeepv@gmail.com>
    > >>>> > > > > > > > wrote:
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >>The alternative foolproof way is to simply not
    > >>>> > > > > > > > provide
    > >>>> > > the
    > >>>> > > > > > > > automatic
    > >>>> > > > > > > >     >> >>download.
    > >>>> > > > > > > >     >> >>
    > >>>> > > > > > > >     >> >>Just document the mirror list URL where the
    > >>>> > > > > > > > templates
    > >>>> > > can
    > >>>> > > > be
    > >>>> > > > > > > > found. I
    > >>>> > > > > > > >     >> >>don't know why people reject this option.
    > >>>> > > > > > > >     >> >>
    > >>>> > > > > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
    > >>>> > > > > > > templates-offline/
    > >>>> > > > > > > >     >> >>
    > >>>> > > > > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
    > >>>> > > > > > > >     >> >><wi...@gmail.com>
    > >>>> > > > > > > >     >> >>wrote:
    > >>>> > > > > > > >     >> >>
    > >>>> > > > > > > >     >> >>> OK. Thanks for the heads up.
    > >>>> > > > > > > >     >> >>>
    > >>>> > > > > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
    > >>>> > > > > > > > chiradeepv@gmail.com>
    > >>>> > > > > > > >     >> wrote:
    > >>>> > > > > > > >     >> >>>
    > >>>> > > > > > > >     >> >>> > Sounds workable. The downloader code in the
    > >>>> > > > > > > > SSVM
    > >>>> > > won't
    > >>>> > > > > > > follow
    > >>>> > > > > > > >     >> >>> > redirects I think.
    > >>>> > > > > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
    > >>>> > > > > > > 5511065fc20787619d
    > >>>> > > > > > > > 9cd0444
    > >>>> > > > > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/
    > >>>> > > storage/template/
    > >>>> > > > > > > >     >> >>> > HttpTemplateDownloader.java#L93
    > >>>> > > > > > > >     >> >>> > https://goo.gl/dSi0r5
    > >>>> > > > > > > >     >> >>> >  Might need to add
    > >>>> > > > > > > >     >> >>> > client.setRedirectStrategy(new
    > >>>> > > LaxRedirectStrategy());
    > >>>> > > > > > > >     >> >>> >
    > >>>> > > > > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will
    > Stevens
    > >>>> > > > > > > >     >> >>> > <ws...@cloudops.com>
    > >>>> > > > > > > >     >> >>> > wrote:
    > >>>> > > > > > > >     >> >>> >
    > >>>> > > > > > > >     >> >>> > > We haven't opened a ticket yet because we
    > >>>> > > > > > > > don't
    > >>>> > > > have a
    > >>>> > > > > > > > strategy
    > >>>> > > > > > > >     >> yet.
    > >>>> > > > > > > >     >> >>> > >
    > >>>> > > > > > > >     >> >>> > > What do you guys think of this:
    > >>>> > > > > > > >     >> >>> > > - We setup a new github repo in the
    > > 'apache'
    > >>>> > > > > > > > org
    > >>>> > > > which
    > >>>> > > > > > > > consists
    > >>>> > > > > > > >     >> >>> > > of a
    > >>>> > > > > > > >     >> >>> > single
    > >>>> > > > > > > >     >> >>> > > file with a list of active/supported
    > >> mirrors.
    > >>>> > > > > > > >     >> >>> > > - I write a small web server, distributed
    > > as
    > >>>> > > > > > > > a
    > >>>> > > > binary,
    > >>>> > > > > > > > which can
    > >>>> > > > > > > >     >> >>> > > be
    > >>>> > > > > > > >     >> >>> > hosted
    > >>>> > > > > > > >     >> >>> > > by ASF Infra.  This web server will query
    > >> the
    > >>>> > > > current
    > >>>> > > > > > list
    > >>>> > > > > > > > of
    > >>>> > > > > > > >     >> >>> > > mirrors
    > >>>> > > > > > > >     >> >>> and
    > >>>> > > > > > > >     >> >>> > > will select one and then do a 302
    > redirect
    > >> to
    > >>>> > > > > > > > that
    > >>>> > > > > > mirror.
    > >>>> > > > > > > >     >> >>> > >
    > >>>> > > > > > > >     >> >>> > > The act of 'choosing' a mirror could be
    > > done
    > >>>> > > > > > > > in a
    > >>>> > > > > number
    > >>>> > > > > > > of
    > >>>> > > > > > > > ways.
    > >>>> > > > > > > >     >> >>> > > - If we want to define an order, then it
    > >>>> > > > > > > > could
    > >>>> > > just
    > >>>> > > > > try
    > >>>> > > > > > > > from the
    > >>>> > > > > > > >     >> >>> > > top of
    > >>>> > > > > > > >     >> >>> > the
    > >>>> > > > > > > >     >> >>> > > list and work its way down.  It would
    > curl
    > >>>> > > > > > > > the
    > >>>> > > > target
    > >>>> > > > > to
    > >>>> > > > > > > > make
    > >>>> > > > > > > >     >> >>> > > sure it
    > >>>> > > > > > > >     >> >>> > gets
    > >>>> > > > > > > >     >> >>> > > a 200 and if it does, it would do a 302
    > >>>> > redirect.
    > >>>> > > > > > > >     >> >>> > > - Or, if we want to distribute the load
    > >>>> > > > > > > > across the
    > >>>> > > > > > > mirrors,
    > >>>> > > > > > > > we
    > >>>> > > > > > > >     >> >>> > > could
    > >>>> > > > > > > >     >> >>> pick
    > >>>> > > > > > > >     >> >>> > > from the list randomly.  Again, doing a
    > > curl
    > >>>> > > > > > > > to
    > >>>> > > > verify
    > >>>> > > > > > the
    > >>>> > > > > > > > mirror
    > >>>> > > > > > > >     >> >>> > > is up
    > >>>> > > > > > > >     >> >>> > and
    > >>>> > > > > > > >     >> >>> > > then doing a redirect.
    > >>>> > > > > > > >     >> >>> > > - If we want to get fancy, we could do a
    > >>>> > > > > > > > reverse
    > >>>> > > IP
    > >>>> > > > > > lookup
    > >>>> > > > > > > > and
    > >>>> > > > > > > >     >> >>> > > try to
    > >>>> > > > > > > >     >> >>> > match
    > >>>> > > > > > > >     >> >>> > > the requester with their closest
    > >> geographical
    > >>>> > > > mirror.
    > >>>> > > > > > > >     >> >>> > >
    > >>>> > > > > > > >     >> >>> > > Thoughts?
    > >>>> > > > > > > >     >> >>> > >
    > >>>> > > > > > > >     >> >>> > > *Will STEVENS*
    > >>>> > > > > > > >     >> >>> > > Lead Developer
    > >>>> > > > > > > >     >> >>> > >
    > >>>> > > > > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
    > >>>> > > > > > > >     >> >>> > >
    > >>>> > > > > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM,
    > Chiradeep
    > >>>> > > Vittal <
    > >>>> > > > > > > >     >> >>> chiradeepv@gmail.com
    > >>>> > > > > > > >     >> >>> > >
    > >>>> > > > > > > >     >> >>> > > wrote:
    > >>>> > > > > > > >     >> >>> > >
    > >>>> > > > > > > >     >> >>> > > > What steps are needed to set up a
    > mirror?
    > >>>> > > > > > > > What
    > >>>> > > > does
    > >>>> > > > > > > Infra
    > >>>> > > > > > > > need
    > >>>> > > > > > > >     >> to do?
    > >>>> > > > > > > >     >> >>> > Has
    > >>>> > > > > > > >     >> >>> > > > anybody filed a ticket with Infra?
    > >>>> > > > > > > >     >> >>> > > >
    > >>>> > > > > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja
    > >>>> > Pullela <
    > >>>> > > > > > > >     >> >>> > > > raja.pullela@accelerite.com>
    > >>>> > > > > > > >     >> >>> > > > wrote:
    > >>>> > > > > > > >     >> >>> > > >
    > >>>> > > > > > > >     >> >>> > > > > Hi will,
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > > > I believe, we didn’t get to close
    > >>>> > > > > > > > ‘getting a
    > >>>> > > > > mirror
    > >>>> > > > > > on
    > >>>> > > > > > > > Apache’
    > >>>> > > > > > > >     >> >>> > because
    > >>>> > > > > > > >     >> >>> > > we
    > >>>> > > > > > > >     >> >>> > > > > needed someone on the Apache Infra
    > side
    > >>>> > > > > > > > to
    > >>>> > > close
    > >>>> > > > > > this.
    > >>>> > > > > > > > BTW,
    > >>>> > > > > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux
    > >>>> > > > > > > > manages
    > >>>> > > > this?)
    > >>>> > > > > > has
    > >>>> > > > > > > >     >> >>> > > > > all/most of
    > >>>> > > > > > > >     >> >>> > the
    > >>>> > > > > > > >     >> >>> > > > > content.  Once we can close on the
    > >> Apache
    > >>>> > > mirror
    > >>>> > > > > for
    > >>>> > > > > > > > hosting
    > >>>> > > > > > > >     >> >>> > > > > the
    > >>>> > > > > > > >     >> >>> > > > content, I
    > >>>> > > > > > > >     >> >>> > > > > can help assist getting the content
    > >> there.
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > > > For now, we have replicated the
    > >>>> > > > > download.cloud.com
    > >>>> > > > > > > > content to
    > >>>> > > > > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
    > >>>> > > > > > > >     >> >>> > > > > Also, we are working on a set of
    > >>>> > > steps/procedure
    > >>>> > > > > to
    > >>>> > > > > > > > help with
    > >>>> > > > > > > >     >> >>> > > > > this change.  I will update everyone
    > in
    > >>>> > > > > > > > about
    > >>>> > > a
    > >>>> > > > > > week’s
    > >>>> > > > > > > > time
    > >>>> > > > > > > >     >> >>> > > > > on the
    > >>>> > > > > > > >     >> >>> > details.
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > > > Best,
    > >>>> > > > > > > >     >> >>> > > > > Raja Pullela
    > >>>> > > > > > > >     >> >>> > > > > Engineering Team,
    > >>>> > > > > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road,
    > Santa
    > >>>> > > > > > > > Clara,
    > >>>> > > > CA,
    > >>>> > > > > > > 95054
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "
    > >>>> > > williamstevens@gmail.com
    > >>>> > > > > on
    > >>>> > > > > > > > behalf of
    > >>>> > > > > > > >     >> >>> > > > > Will Stevens" <
    > williamstevens@gmail.com
    > >>>> > > > > > > > on
    > >>>> > > > behalf
    > >>>> > > > > > of
    > >>>> > > > > > > >     >> >>> > wstevens@cloudops.com>
    > >>>> > > > > > > >     >> >>> > > > > wrote:
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > > > unfortunately the template mirror
    > >>>> > > > > > > > conversation
    > >>>> > > > got
    > >>>> > > > > > > > caught up
    > >>>> > > > > > > >     >> >>> > > > > in
    > >>>> > > > > > > >     >> >>> > details
    > >>>> > > > > > > >     >> >>> > > > and
    > >>>> > > > > > > >     >> >>> > > > > nobody took the lead on implementing
    > a
    > >>>> > > solution.
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > > > citrix has been pinging me every
    > couple
    > >>>> > > > > > > > months
    > >>>> > > > to
    > >>>> > > > > > say
    > >>>> > > > > > > > 'dude,
    > >>>> > > > > > > >     >> >>> > > > > we
    > >>>> > > > > > > >     >> >>> need
    > >>>> > > > > > > >     >> >>> > to
    > >>>> > > > > > > >     >> >>> > > > > remove the dependency on
    > >>>> > > > > > > > download.citrix.com
    > >>>> > > ',
    > >>>> > > > > but
    > >>>> > > > > > i
    > >>>> > > > > > > > have not
    > >>>> > > > > > > >     >> >>> > > > > had
    > >>>> > > > > > > >     >> >>> > the
    > >>>> > > > > > > >     >> >>> > > > > cycles to get in and solve the
    > problem.
    > >>>> > > > > > > > the
    > >>>> > > > > > shutdown
    > >>>> > > > > > > > of that
    > >>>> > > > > > > >     >> >>> > > > > is
    > >>>> > > > > > > >     >> >>> > > imminent
    > >>>> > > > > > > >     >> >>> > > > > right now, so we need to solve it
    > asap.
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > > > *Will STEVENS*
    > >>>> > > > > > > >     >> >>> > > > > Lead Developer
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM,
    > Paul
    > >>>> > Angus <
    > >>>> > > > > > > >     >> >>> > paul.angus@shapeblue.com
    > >>>> > > > > > > >     >> >>> > > >
    > >>>> > > > > > > >     >> >>> > > > > wrote:
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > > > > Hi Nathan,
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > Ideally, if you put the template
    > >>>> > > > > > > > location in
    > >>>> > > > (or
    > >>>> > > > > > use
    > >>>> > > > > > > a
    > >>>> > > > > > > >     >> >>> > > > > > template
    > >>>> > > > > > > >     >> >>> > > defined
    > >>>> > > > > > > >     >> >>> > > > > > in)  test_data.py then the actual
    > >>>> > > > > > > > location
    > >>>> > > can
    > >>>> > > > > be
    > >>>> > > > > > > >     >> >>> > > > > > overridden by
    > >>>> > > > > > > >     >> >>> > > anyone
    > >>>> > > > > > > >     >> >>> > > > > > testing.
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > For Trillian, we've copied all of
    > the
    > >>>> > > > templates
    > >>>> > > > > > that
    > >>>> > > > > > > > people
    > >>>> > > > > > > >     >> >>> > > > > > have
    > >>>> > > > > > > >     >> >>> > > define
    > >>>> > > > > > > >     >> >>> > > > > to
    > >>>> > > > > > > >     >> >>> > > > > > a local repo and then replace the
    > > URLs
    > >>>> > > > > > > > in test_data.py to
    > >>>> > > > > > > >     >> >>> > > > > > reduce
    > >>>> > > > > > > >     >> >>> > > > > bandwidth
    > >>>> > > > > > > >     >> >>> > > > > > use and download times.
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > Ie:
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > >             "bootableIso":
    > >>>> > > > > > > >     >> >>> > > > > >                 {
    > >>>> > > > > > > >     >> >>> > > > > >                     "displaytext":
    > >> "Test
    > >>>> > > > > Bootable
    > >>>> > > > > > > > ISO",
    > >>>> > > > > > > >     >> >>> > > > > >                     "name":
    > > "testISO",
    > >>>> > > > > > > >     >> >>> > > > > >                     "bootable":
    > True,
    > >>>> > > > > > > >     >> >>> > > > > >                     "ispublic":
    > > False,
    > >>>> > > > > > > >     >> >>> > > > > >                     "url": "{{
    > >>>> > > > > > > marvin_images_location
    > >>>> > > > > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
    > >>>> > > > > > > >     >> >>> > > > > >                     "ostype":
    > 'Other
    > >>>> Linux
    > >>>> > > > > > > (64-bit)',
    > >>>> > > > > > > >     >> >>> > > > > >                     "mode":
    > >>>> > 'HTTP_DOWNLOAD'
    > >>>> > > > > > > >     >> >>> > > > > >         },
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > I thought that we had come up with
    > a
    > >>>> > > solution
    > >>>> > > > > for
    > >>>> > > > > > > >     >> >>> > download.cloud.com
    > >>>> > > > > > > >     >> >>> > > ,
    > >>>> > > > > > > >     >> >>> > > > by
    > >>>> > > > > > > >     >> >>> > > > > > having a mirrorlist hosted in
    > >> Community
    > >>>> > > Apache
    > >>>> > > > > > > > 'space' with
    > >>>> > > > > > > >     >> >>> anyone
    > >>>> > > > > > > >     >> >>> > > able
    > >>>> > > > > > > >     >> >>> > > > > to
    > >>>> > > > > > > >     >> >>> > > > > > out themselves forward as a mirror.
    > >>>> > > > > > > >     >> >>> > > > > > But I must admit I lost track of
    > >>>> > > > > > > > whether
    > >>>> > > > anyone
    > >>>> > > > > > made
    > >>>> > > > > > > > the
    > >>>> > > > > > > >     >> >>> requisite
    > >>>> > > > > > > >     >> >>> > > > > changes
    > >>>> > > > > > > >     >> >>> > > > > > in code....
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > Kind regards,
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > Paul Angus
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > paul.angus@shapeblue.com
    > >>>> > > > > > > >     >> >>> > > > > > www.shapeblue.com
    > >>>> > > > > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden,
    > >> London
    > >>>> > > WC2N
    > >>>> > > > > > 4HSUK
    > >>>> > > > > > > >     >> >>> > > > > > @shapeblue
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > -----Original Message-----
    > >>>> > > > > > > >     >> >>> > > > > > From: williamstevens@gmail.com
    > >>>> > > > > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
    > >>>> > > > > > > >     >> >>> > On
    > >>>> > > > > > > >     >> >>> > > > > > Behalf Of Will Stevens
    > >>>> > > > > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
    > >>>> > > > > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
    > >>>> > > > > > > >     >> >>> > > > > > Subject: Re: Modern template
    > hosting
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > this is a hard questions.  in
    > > general,
    > >>>> > > > > > > > we
    > >>>> > > > should
    > >>>> > > > > > be
    > >>>> > > > > > > > setting
    > >>>> > > > > > > >     >> >>> > > > > > up a
    > >>>> > > > > > > >     >> >>> > > mirror
    > >>>> > > > > > > >     >> >>> > > > > on
    > >>>> > > > > > > >     >> >>> > > > > > some cloudstack/apache domain and
    > > then
    > >>>> > > mirror
    > >>>> > > > to
    > >>>> > > > > > > other
    > >>>> > > > > > > >     >> >>> > > > > > provided
    > >>>> > > > > > > >     >> >>> > > > > templates.
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > we MUST come up with a solution to
    > >>>> > deprecate
    > >>>> > > > > > > >     >> >>> > > > > > 'download.cloud.com
    > >>>> > > > > > > >     >> >>> ',
    > >>>> > > > > > > >     >> >>> > > > that
    > >>>> > > > > > > >     >> >>> > > > > > is going to be going away any day
    > > now.
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > i don't know the right way to solve
    > >>>> > > > > > > > this to
    > >>>> > > be
    > >>>> > > > > > > > honest, but
    > >>>> > > > > > > >     >> >>> > > > > > if you
    > >>>> > > > > > > >     >> >>> > > have
    > >>>> > > > > > > >     >> >>> > > > > > ideas, i am willing to help.
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > *Will STEVENS*
    > >>>> > > > > > > >     >> >>> > > > > > Lead Developer
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM,
    > >>>> > > > > > > > Nathan
    > >>>> > > > > Johnson <
    > >>>> > > > > > > >     >> >>> njohnson@ena.com
    > >>>> > > > > > > >     >> >>> > >
    > >>>> > > > > > > >     >> >>> > > > > wrote:
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > > So not to re-open a can of worms,
    > >> but
    > >>>> > > > > > > > I’m
    > >>>> > > > in a
    > >>>> > > > > > > > situation
    > >>>> > > > > > > >     >> >>> > > > > > > where
    > >>>> > > > > > > >     >> >>> I
    > >>>> > > > > > > >     >> >>> > > need
    > >>>> > > > > > > >     >> >>> > > > > > > to come up with a Marvin
    > component
    > >>>> > > > > > > > test
    > >>>> > > that
    > >>>> > > > > > > > depends on a
    > >>>> > > > > > > >     >> >>> > template
    > >>>> > > > > > > >     >> >>> > > > > > > based on a kernel that’s
    > relatively
    > >>>> > > > > > > > new,
    > >>>> > > > i.e.,
    > >>>> > > > > > > > newer than
    > >>>> > > > > > > >     >> >>> Centos
    > >>>> > > > > > > >     >> >>> > > 5.3
    > >>>> > > > > > > >     >> >>> > > > /
    > >>>> > > > > > > >     >> >>> > > > > > Ubuntu 10.04 .
    > >>>> > > > > > > >     >> >>> > > > > > > I see openvm.eu has a suitable
    > >>>> > > > > > > > template
    > >>>> > > > > (Ubuntu
    > >>>> > > > > > > > 16.0.4
    > >>>> > > > > > > >     >> >>> > > > > > > for
    > >>>> > > > > > > >     >> >>> KVM),
    > >>>> > > > > > > >     >> >>> > > but
    > >>>> > > > > > > >     >> >>> > > > > > > from looking at the thread
    > >> "Migrating
    > >>>> > > > > CloudStack
    > >>>> > > > > > > > content
    > >>>> > > > > > > >     >> >>> > > > > > > from download.cloud.com” it
    > looks
    > >>>> > > > > > > > like
    > >>>> > > > there
    > >>>> > > > > is
    > >>>> > > > > > > >     >> >>> > > > > > > resistance to using
    > >>>> > > > > > > >     >> >>> > > this
    > >>>> > > > > > > >     >> >>> > > > at
    > >>>> > > > > > > >     >> >>> > > > > > > least for hosting system vm
    > >> templates
    > >>>> > > > > > > > over
    > >>>> > > > > > > concerns
    > >>>> > > > > > > > of
    > >>>> > > > > > > >     >> >>> > neutrality.
    > >>>> > > > > > > >     >> >>> > > > > > > Would this be suitable for a
    > >>>> > > > > > > > component
    > >>>> > > test?
    > >>>> > > > > If
    > >>>> > > > > > > > not,
    > >>>> > > > > > > >     >> >>> > > > > > > what is a
    > >>>> > > > > > > >     >> >>> > > > > “blessed”
    > >>>> > > > > > > >     >> >>> > > > > > > template location?
    > >>>> > > > > > > >     >> >>> > > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > > Thanks in advance!
    > >>>> > > > > > > >     >> >>> > > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > > Nathan Johnson
    > >>>> > > > > > > >     >> >>> > > > > > > R&D Engineer
    > >>>> > > > > > > >     >> >>> > > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite
    > 12
    > >>>> > > > Nashville,
    > >>>> > > > > TN
    > >>>> > > > > > > > 37211
    > >>>> > > > > > > >     >> >>> > > > > > > General Office: 615-312-6000
    > >>>> > > > > > > >     >> >>> > > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > > website | blog | support
    > >>>> > > > > > > >     >> >>> > > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > >
    > >>>> > > > > > > >     >> >>> > > > > > >
    > >>>> > > > > > > >     >> >>> > > > > >
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > > > DISCLAIMER
    > >>>> > > > > > > >     >> >>> > > > > ==========
    > >>>> > > > > > > >     >> >>> > > > > This e-mail may contain privileged
    > and
    > >>>> > > > > confidential
    > >>>> > > > > > > >     >> >>> > > > > information
    > >>>> > > > > > > >     >> >>> which
    > >>>> > > > > > > >     >> >>> > > is
    > >>>> > > > > > > >     >> >>> > > > > the property of Accelerite, a
    > > Persistent
    > >>>> > > Systems
    > >>>> > > > > > > > business. It
    > >>>> > > > > > > >     >> >>> > > > > is
    > >>>> > > > > > > >     >> >>> > > intended
    > >>>> > > > > > > >     >> >>> > > > > only for the use of the individual or
    > >>>> > > > > > > > entity
    > >>>> > > to
    > >>>> > > > > > which
    > >>>> > > > > > > > it is
    > >>>> > > > > > > >     >> >>> > addressed.
    > >>>> > > > > > > >     >> >>> > > If
    > >>>> > > > > > > >     >> >>> > > > > you are not the intended recipient,
    > you
    > >>>> > > > > > > > are
    > >>>> > > not
    > >>>> > > > > > > > authorized to
    > >>>> > > > > > > >     >> >>> > > > > read,
    > >>>> > > > > > > >     >> >>> > > > retain,
    > >>>> > > > > > > >     >> >>> > > > > copy, print, distribute or use this
    > >>>> message.
    > >>>> > > If
    > >>>> > > > > you
    > >>>> > > > > > > have
    > >>>> > > > > > > >     >> >>> > > > > received
    > >>>> > > > > > > >     >> >>> > this
    > >>>> > > > > > > >     >> >>> > > > > communication in error, please notify
    > >> the
    > >>>> > > sender
    > >>>> > > > > and
    > >>>> > > > > > > > delete
    > >>>> > > > > > > >     >> >>> > > > > all
    > >>>> > > > > > > >     >> >>> > copies
    > >>>> > > > > > > >     >> >>> > > of
    > >>>> > > > > > > >     >> >>> > > > > this message. Accelerite, a
    > Persistent
    > >>>> > > > > > > > Systems
    > >>>> > > > > > > business
    > >>>> > > > > > > > does
    > >>>> > > > > > > >     >> >>> > > > > not
    > >>>> > > > > > > >     >> >>> > accept
    > >>>> > > > > > > >     >> >>> > > > any
    > >>>> > > > > > > >     >> >>> > > > > liability for virus infected mails.
    > >>>> > > > > > > >     >> >>> > > > >
    > >>>> > > > > > > >     >> >>> > > >
    > >>>> > > > > > > >     >> >>> > >
    > >>>> > > > > > > >     >> >>> >
    > >>>> > > > > > > >     >> >>>
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >abhinandan.prateek@shapeblue.com
    > >>>> > > > > > > >     >> >www.shapeblue.com
    > >>>> > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N
    > > 4HSUK
    > >>>> > > > > @shapeblue
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >> >
    > >>>> > > > > > > >     >>
    > >>>> > > > > > > >     >> abhinandan.prateek@shapeblue.com
    > >>>> > > > > > > >     >> www.shapeblue.com
    > >>>> > > > > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N
    > 4HSUK
    > >>>> > > > > > > >     >> @shapeblue
    > >>>> > > > > > > >     >>
    > >>>> > > > > > > >     >>
    > >>>> > > > > > > >     >>
    > >>>> > > > > > > >     >>
    > >>>> > > > > > > >     >
    > >>>> > > > > > > >
    > >>>> > > > > > > >
    > >>>> > > > > > > >
    > >>>> > > > > > > > daan.hoogland@shapeblue.com
    > >>>> > > > > > > > www.shapeblue.com
    > >>>> > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
    > >>>> > > > > > > > @shapeblue
    > >>>> > > > > > > >
    > >>>> > > > > > > >
    > >>>> > > > > > > >
    > >>>> > > > > > > >
    > >>>> > > > > > >
    > >>>> > > > > > >
    > >>>> > > > > > > --
    > >>>> > > > > > > Rafael Weingärtner
    > >>>> > > > > > >
    > >>>> > > > > >
    > >>>> > > > > >
    > >>>> > > > > >
    > >>>> > > > > > --
    > >>>> > > > > > Rafael Weingärtner
    > >>>> > > > > >
    > >>>> > > > >
    > >>>> > > >
    > >>>> > >
    > >>>> >
    > >>>>
    >
    >
    


daan.hoogland@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue

Re: Modern template hosting

Posted by Will Stevens <wi...@gmail.com>.

I will be guessing and we will see what I can come up with. :)

On Mar 3, 2017 7:09 PM, "Erik Weber" <te...@gmail.com> wrote:

> Feel free to do it, I wouldn't know where to begin to ask such a
> question tbh :-)
>
> --
> Erik
>
> On Sat, Mar 4, 2017 at 12:51 AM, Will Stevens <wi...@gmail.com>
> wrote:
> > Erik, yes I think this is the right approach and covers the main problem
> > case. I can start the conversation with infra unless you would like to.
> Let
> > me know.
> >
> > We did get some statistics from the other mirrors last year, but I don't
> > think we ever had visibility into the cloud.com repository traffic.
> >
> > On Mar 3, 2017 6:42 PM, "Erik Weber" <te...@gmail.com> wrote:
> >
> > It'll most likely only be an issue in the case where a mirror goes
> > down without any immediate chance of getting back up, right?
> > Could we check with ASF Infra beforehand if such case is OK to warrant
> > an urgent Jira ticket to resolve - should it happen?
> >
> > We can document all the mirrors and provide the howtos for manually
> > downloading and seeding the image as a backup.
> >
> > Do we have any traffic statistics for the current solution, or is that
> > not available at S3?
> >
> > --
> > Erik
> >
> > On Sat, Mar 4, 2017 at 12:28 AM, Will Stevens <wi...@gmail.com>
> > wrote:
> >> Yes Erik, I agree with you. The only thing that could add complexity in
> >> this is the fact that we don't control the domain, the ASF does (from
> what
> >> I understand).  Do we expect this to be managed by ASF infra or would
> > there
> >> be another way? Ideally the domain is community controlled so we can
> adapt
> >> in case of change without long delays.
> >>
> >> I am 100% in agreement that this is the shortest path and likely to be
> > good
> >> enough.
> >>
> >> On Mar 3, 2017 6:19 PM, "Erik Weber" <te...@gmail.com> wrote:
> >>
> >> IMHO; implement it as simple as possible, use DNS RR and assume/expect
> >> mirror hosts to use the same path.
> >>
> >>
> >> Yes I know; DNS RR isn't bulletproof, but as already mentioned in this
> >> thread - mirrors rarely go down and they aren't used all that often
> >> (primarily seeding).
> >> We can keep a low TTL on the record so that we're able to remove a
> >> mirror that goes down for a significant period of time (more than
> >> hours).
> >>
> >> And yes I know, some operators might find it a burden having to set up
> >> a new vhost with another document root, but tbh if they don't care
> >> enough to do it we'll manage without that mirror.
> >>
> >> --
> >> Erik
> >>
> >> On Sat, Mar 4, 2017 at 12:09 AM, Will Stevens <williamstevens@gmail.com
> >
> >> wrote:
> >>> I agree with Paul. Look at the list of things they have to
> >>> learn/master/care about. We don't want to add to that list, we want to
> >>> remove from that list. Make the project/product more accessible. Reduce
> >> the
> >>> barrier to entry...
> >>>
> >>> On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <ch...@gmail.com>
> wrote:
> >>>
> >>>> Seriously?
> >>>> apt-get install apache2
> >>>>
> >>>> To install CloudStack, they need to know
> >>>> - DB installation, perhaps some SQL
> >>>> - VLAN configuration on their switches
> >>>> - Ins-and-outs between port forwarding, static ips,
> >>>> - NFS
> >>>> - package management
> >>>> - VHDs, qcow2. vmdk
> >>>> - hyperviosrs
> >>>> - and on and on.
> >>>>
> >>>> If they can figure all that out and not figure out a web server, they
> > can
> >>>> always come to the mailing list.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <pa...@shapeblue.com>
> >>>> wrote:
> >>>>
> >>>> > The issue is not with supporting highly experienced cloud operators.
> > It
> >>>> is
> >>>> > an issue for new users and other 'relatively' inexperienced
> operators.
> >>>> >
> >>>> > I have helped enough newbies and cloud operators who have been
> running
> >>>> > their cloud for a while to know that the barriers to entry are too
> > high
> >>>> as
> >>>> > they are. And telling anyone that they need to create a web server
> so
> >>>> that
> >>>> > they can add their initial template to get started or in order to
> >> create
> >>>> a
> >>>> > new zone, just isn't going to fly.
> >>>> >
> >>>> > I'm a huge advocate of the 'download.cloudstack.org' endpoint which
> > the
> >>>> > community can add/remove mirrors to or from for system VMs or
> built-in
> >>>> > templates.  Can the same system be used for binary repos ? although
> >> there
> >>>> > is an added complication of redist vs no-redist there...
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > paul.angus@shapeblue.com
> >>>> > www.shapeblue.com
> >>>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >>>> > @shapeblue
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > -----Original Message-----
> >>>> > From: Chiradeep Vittal [mailto:chiradeepv@gmail.com]
> >>>> > Sent: 03 March 2017 18:28
> >>>> > To: dev <de...@cloudstack.apache.org>
> >>>> > Subject: Re: Modern template hosting
> >>>> >
> >>>> > I do feel like this is early optimization. Mirrors rarely fail. I'd
> >>>> expect
> >>>> > a single web server hosted on Apache Infra without any monitors to
> > fail
> >>>> > more often than a mirror. We already expect Wido's systemvm
> repository
> >> to
> >>>> > be up all the time. And it has been. Similarly, I don't believe
> Nux's
> >>>> > repository has ever been down. And if Accelerite wants to host on
> S3,
> >>>> that
> >>>> > one is pretty solid as well.
> >>>> >
> >>>> > This is an infrequent operation in a cloud. After the cloud is
> >> installed,
> >>>> > the download servers are only needed for a new zone. If we trust the
> >> user
> >>>> > to run a cloud, surely he/she can run a web server to serve some
> >> built-in
> >>>> > templates. And if her cloud is successful, she needs to figure out
> how
> >> to
> >>>> > host her templates anyway and not rely on 3rd parties.
> >>>> >
> >>>> >
> >>>> > On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <wstevens@cloudops.com
> >
> >>>> > wrote:
> >>>> >
> >>>> > > 1) If the legacy implementations do not support redirects, that
> does
> >>>> > > cause a problem.  A potential solution in that case is to have the
> >> web
> >>>> > > server actually proxy the download, but that is not ideal and I
> > would
> >>>> > > like to avoid it if possible.  Thanks for bringing that up
> > Chiradeep.
> >>>> > >
> >>>> > > 2) I think we need to have a single URL which people can target.
> >> Once
> >>>> > > they make the switch to the new URL, we want the implementation to
> > be
> >>>> > > able to handle mirror failures without affecting the end client.
> We
> >>>> > > want to avoid the situation where an ACS user will ever have to
> >> change
> >>>> > > this URL more than once.  Mirror failures SHOULD NOT affect the
> ACS
> >>>> > > users assuming there is still at least one mirror who can serve
> the
> >>>> > requested resource.
> >>>> > >
> >>>> > > These are obviously my personal opinions and others will probably
> >> have
> >>>> > > differing opinions.
> >>>> > >
> >>>> > > *Will STEVENS*
> >>>> > > Lead Developer
> >>>> > >
> >>>> > > <https://goo.gl/NYZ8KK>
> >>>> > >
> >>>> > > On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal
> >>>> > > <ch...@gmail.com>
> >>>> > > wrote:
> >>>> > >
> >>>> > > > 1. If you are targeting legacy installations, they are not able
> to
> >>>> > > > follow redirects. The line of code that added this capability
> was
> >>>> > > > added on
> >>>> > > 11/16.
> >>>> > > > 2. If you trust the users to edit the database to change the
> URL,
> >>>> > > > you can trust them to change it to anything. Just document a
> known
> >>>> > > > list of good template locations.
> >>>> > > >
> >>>> > > > On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <
> > wstevens@cloudops.com
> >>>
> >>>> > > > wrote:
> >>>> > > >
> >>>> > > > > So the main issue I see with this is the following.
> >>>> > > > >
> >>>> > > > > This implementation is designed to target legacy installations
> >>>> > > > > which
> >>>> > > will
> >>>> > > > > be affected when download.cloud.com disappears.  These people
> >> will
> >>>> > > need
> >>>> > > > to
> >>>> > > > > make a DB change to replace the 'download.cloud.com' with
> some
> >>>> > > > > other
> >>>> > > url
> >>>> > > > > (maybe 'download.cloudstack.org').  Once the DB has been
> > updated,
> >>>> > > > > we
> >>>> > > can
> >>>> > > > > not expect anything else of the client, they need to be able
> to
> >>>> > > > > just continue operation as they were without any need to
> rebuild
> >>>> > > > > or upgrade their ACS.
> >>>> > > > >
> >>>> > > > > If we try to force the decision to the client, then we break
> the
> >>>> > > > > legacy implementations.  Since the SSVM is likely going to be
> > the
> >>>> > > > > client in
> >>>> > > some
> >>>> > > > > cases, and since it does not already have the logic to handle
> > the
> >>>> > > > > 300 approach correctly, I feel like it is not a viable
> solution
> >>>> > > > > for the
> >>>> > > > legacy
> >>>> > > > > deployments.
> >>>> > > > >
> >>>> > > > > In addition to that.  I don't think the client is in any
> better
> >>>> > > position
> >>>> > > > to
> >>>> > > > > make the mirror decision than I am.  I am likely able to make
> a
> >>>> > > > > more 'educated' decision than the client would be able to
> > because
> >>>> > > > > I can do different tests on the endpoint(s) before making a
> > final
> >>>> > decision.
> >>>> > > > >
> >>>> > > > > Does this make sense or am I missing something obvious?
> >>>> > > > >
> >>>> > > > > *Will STEVENS*
> >>>> > > > > Lead Developer
> >>>> > > > >
> >>>> > > > > <https://goo.gl/NYZ8KK>
> >>>> > > > >
> >>>> > > > > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
> >>>> > > > > rafaelweingartner@gmail.com> wrote:
> >>>> > > > >
> >>>> > > > > > I think I understood you, but I did not understand you.
> >>>> > > > > >
> >>>> > > > > > Let me see if I can create an example to illustrate how we
> >> could
> >>>> > > > > > do
> >>>> > > > using
> >>>> > > > > > the HTTP 300 code.
> >>>> > > > > >
> >>>> > > > > > The HTTP 300 code indicates clients that he/she/it has
> > multiple
> >>>> > > choices
> >>>> > > > > to
> >>>> > > > > > access the requested resource. If the request is a HEAD,
> this
> >> is
> >>>> > > > > > the
> >>>> > > > only
> >>>> > > > > > thing that the server returns. If the request if a GET, then
> >> the
> >>>> > > server
> >>>> > > > > > returns the choices for the client. The response type is
> >> defined
> >>>> > > > > > on
> >>>> > > the
> >>>> > > > > > server based on the Content-Type, user agent and others.
> >>>> > > > > >
> >>>> > > > > > Let’s say the server receives a GET request and informs the
> >>>> > > > content-type
> >>>> > > > > as
> >>>> > > > > > JSON (application/JSON). Your application could get the
> > mirrors
> >>>> > > > > > list,
> >>>> > > > do
> >>>> > > > > > the appending process for the requested resource path, and
> > then
> >>>> > > return
> >>>> > > > a
> >>>> > > > > > JSON array with the possible mirror the client can use.
> Then,
> >>>> > > > > > the
> >>>> > > > client
> >>>> > > > > > gets this list and does the processing required to select a
> >>>> mirror.
> >>>> > > > > >
> >>>> > > > > > Now let’s say that a human access the link using a browser.
> >>>> > > > > > Then, the server should receive something like (text/plain)
> as
> >>>> > > > > > the
> >>>> > > content-type;
> >>>> > > > I
> >>>> > > > > > think depending on the browser this field may differ a
> little
> >>>> > > > > > (it
> >>>> > > would
> >>>> > > > > > require some checking). Anyways, the server detects that it
> is
> >> a
> >>>> > > > “human”
> >>>> > > > > > requesting the resource, then we could serve an HTML page
> with
> >> a
> >>>> > > > > Javascript
> >>>> > > > > > that uses the mirror list. This Javascript could do some
> >>>> > > > > > checking and choose the best mirror for that specific.
> >>>> > > > > >
> >>>> > > > > > Why do I think this approach is interesting?
> >>>> > > > > > The application you developed would be used only to retrieve
> >>>> > > > > > valid
> >>>> > > and
> >>>> > > > > > trusted sources of system VM images (a trusted repository
> for
> >>>> > > > > > mirrors
> >>>> > > > > that
> >>>> > > > > > we as a community vouch for).  Also, we shift the decision
> >>>> > > > > > process regarding mirrors from the server to the client.
> Then,
> >>>> > > > > > it is up to
> >>>> > > > > clients
> >>>> > > > > > to select mirrors, and not up to us (or some of our
> >>>> > implementations).
> >>>> > > > > >
> >>>> > > > > > Did this help?
> >>>> > > > > > BTW: I am not saying I am against the way you proposed,
> which
> >>>> > > > > > would
> >>>> > > > work
> >>>> > > > > > fine. It is merely a suggestion using a different
> perspective.
> >>>> > > > > >
> >>>> > > > > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
> >>>> > > > williamstevens@gmail.com>
> >>>> > > > > > wrote:
> >>>> > > > > >
> >>>> > > > > > > @rafael: in general, I don't think the client should ever
> be
> >>>> > > making a
> >>>> > > > > > > choice about a mirror. We have to assume we are working
> with
> >> a
> >>>> > > > scripted
> >>>> > > > > > > application and anything hard coded in that implementation
> > on
> >>>> > > > > > > the
> >>>> > > > > client
> >>>> > > > > > > side is a risk.
> >>>> > > > > > >
> >>>> > > > > > > I may not be understanding the use of the 300 approach,
> so I
> >>>> > > > > > > will
> >>>> > > > > > research
> >>>> > > > > > > it to see if I can make it fit.
> >>>> > > > > > >
> >>>> > > > > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
> >>>> > > > > > rafaelweingartner@gmail.com>
> >>>> > > > > > > wrote:
> >>>> > > > > > >
> >>>> > > > > > > Will great job.
> >>>> > > > > > >
> >>>> > > > > > > I had the same doubt as Daan.
> >>>> > > > > > >
> >>>> > > > > > > Have you considered using HTTP 300 working mode? Then, we
> >>>> > > > > > > could let
> >>>> > > > the
> >>>> > > > > > > client decide which mirror is the best(closest?). Section
> >>>> > “10.3.1”
> >>>> > > of
> >>>> > > > > [1]
> >>>> > > > > > > talks about it; this would put the pressure on deciding
> upon
> >> a
> >>>> > > mirror
> >>>> > > > > on
> >>>> > > > > > > the client side, but I think it would be fair.
> >>>> > > > > > >
> >>>> > > > > > > [1] https://www.w3.org/Protocols/
> rfc2616/rfc2616-sec10.html
> >>>> > > > > > >
> >>>> > > > > > > PS: You really like very short variables names!
> >>>> > > > > > >
> >>>> > > > > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
> >>>> > > > > > daan.hoogland@shapeblue.com
> >>>> > > > > > > >
> >>>> > > > > > > wrote:
> >>>> > > > > > >
> >>>> > > > > > > > Nice little thing Will,
> >>>> > > > > > > >
> >>>> > > > > > > > One question: if I read the code correctly it ‘go’es and
> >>>> > > > > > > > tries
> >>>> > > all
> >>>> > > > > > > mirrors
> >>>> > > > > > > > at once and whichever responses first is redirected to
> the
> >>>> > > client.
> >>>> > > > > This
> >>>> > > > > > > > might well be the same every time. This might be the one
> >>>> > > > > > > > closest
> >>>> > > to
> >>>> > > > > the
> >>>> > > > > > > > mirror302 and not to the client. Is that correct or did
> I
> >>>> > > > > > > > miss a
> >>>> > > > > > weighing
> >>>> > > > > > > > algorithm hidden in there?
> >>>> > > > > > > >
> >>>> > > > > > > > Good coding,
> >>>> > > > > > > >
> >>>> > > > > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org>
> > wrote:
> >>>> > > > > > > >
> >>>> > > > > > > >     Hey All,
> >>>> > > > > > > >     Please review this repo and tell me what you think:
> >>>> > > > > > > >     https://github.com/swill/mirror302
> >>>> > > > > > > >
> >>>> > > > > > > >     Let me know if you have questions or you would like
> me
> >>>> > > > > > > > to
> >>>> > > make
> >>>> > > > > > > changes
> >>>> > > > > > > >     before I bring this topic up with ASF Infra and
> open a
> >>>> > > > > > > > ticket
> >>>> > > > to
> >>>> > > > > > get
> >>>> > > > > > > > this
> >>>> > > > > > > >     implemented.
> >>>> > > > > > > >
> >>>> > > > > > > >     Cheers,
> >>>> > > > > > > >
> >>>> > > > > > > >     *Will Stevens*
> >>>> > > > > > > >
> >>>> > > > > > > >
> >>>> > > > > > > >
> >>>> > > > > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> >>>> > > > > > wstevens@cloudops.com
> >>>> > > > > > > >
> >>>> > > > > > > > wrote:
> >>>> > > > > > > >
> >>>> > > > > > > >     > I am building a short term solution right now.  I
> >>>> > > > > > > > hopefully
> >>>> > > > > will
> >>>> > > > > > > > have the
> >>>> > > > > > > >     > building blocks in place by the weekend so we can
> >>>> > > > > > > > start
> >>>> > > > working
> >>>> > > > > > > with
> >>>> > > > > > > > Infra
> >>>> > > > > > > >     > to get it in place.
> >>>> > > > > > > >     >
> >>>> > > > > > > >     > We will have a web server which we can point
> >> something
> >>>> > like
> >>>> > > > > > > >     > downloads.cloudstack.org (or whatever url) at.
> Then
> >>>> > > > > > > > we
> >>>> > > will
> >>>> > > > > > have
> >>>> > > > > > > > an ASF
> >>>> > > > > > > >     > repo which tracks a mirror list and exposes a
> static
> >>>> > > > > > > > site
> >>>> > > to
> >>>> > > > be
> >>>> > > > > > > able
> >>>> > > > > > > > to
> >>>> > > > > > > >     > query the mirror list.
> >>>> > > > > > > >     >
> >>>> > > > > > > >     > The web server will receive a request and will do
> a
> >>>> > > > > > > > 302
> >>>> > > > > redirect
> >>>> > > > > > to
> >>>> > > > > > > > the
> >>>> > > > > > > >     > appropriate resource on one of the mirrors.
> >>>> > > > > > > >     >
> >>>> > > > > > > >     > This gets us started.  Legacy environments will
> have
> >>>> > > > > > > > to do
> >>>> > > a
> >>>> > > > DB
> >>>> > > > > > > > change to
> >>>> > > > > > > >     > change from 'download.cloud.com' to '
> >>>> > > > downloads.cloudstack.org'
> >>>> > > > > > (or
> >>>> > > > > > > >     > whatever), but otherwise it should be pretty
> simple.
> >>>> > > > > > > >     >
> >>>> > > > > > > >     > We can improve how we deliver templates going
> >> forward,
> >>>> > > > > > > > but
> >>>> > > > this
> >>>> > > > > > > > obviously
> >>>> > > > > > > >     > requires some discussion still.  I will try to buy
> > us
> >>>> > > > > > > > some
> >>>> > > > time
> >>>> > > > > > > with
> >>>> > > > > > > > an
> >>>> > > > > > > >     > implementation which solves our problems today...
> >>>> > > > > > > >     >
> >>>> > > > > > > >     > *Will STEVENS*
> >>>> > > > > > > >     > Lead Developer
> >>>> > > > > > > >     >
> >>>> > > > > > > >     > <https://goo.gl/NYZ8KK>
> >>>> > > > > > > >     >
> >>>> > > > > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan
> Prateek <
> >>>> > > > > > > >     > abhinandan.prateek@shapeblue.com> wrote:
> >>>> > > > > > > >     >
> >>>> > > > > > > >     >>
> >>>> > > > > > > >     >> Initial seeding is a manual step and that is not
> >>>> > > > > > > > going to
> >>>> > > > > change
> >>>> > > > > > > in
> >>>> > > > > > > > near
> >>>> > > > > > > >     >> future. A handy list of official places from
> where
> >>>> > > > > > > > these
> >>>> > > > > > templates
> >>>> > > > > > > > can be
> >>>> > > > > > > >     >> downloaded won’t harm this part of the
> > installation.
> >>>> > > > > > > > Just
> >>>> > > > > allow
> >>>> > > > > > > > anyone
> >>>> > > > > > > >     >> credible who follows democratic process to
> publish
> >>>> > > > > > > > their
> >>>> > > > > > templates
> >>>> > > > > > > > with
> >>>> > > > > > > >     >> apache’s blessings.
> >>>> > > > > > > >     >>
> >>>> > > > > > > >     >>
> >>>> > > > > > > >     >> Coming to urls hard coded in db, the popular
> >>>> > > > > > > > suggestion is
> >>>> > > > to
> >>>> > > > > > > point
> >>>> > > > > > > > them
> >>>> > > > > > > >     >> to mirrors. This looks good but this is going to
> >> take
> >>>> > > > > > > > some
> >>>> > > > > > effort
> >>>> > > > > > > > that has
> >>>> > > > > > > >     >> not materialised yet. Once this change is made it
> >>>> > > > > > > > will be
> >>>> > > > > going
> >>>> > > > > > in
> >>>> > > > > > > > some
> >>>> > > > > > > >     >> future release like 4.11; the problem still
> remains
> >>>> > > > > > > > for
> >>>> > > the
> >>>> > > > > > > > releases that
> >>>> > > > > > > >     >> are already out there. In case access to
> >>>> > > download.cloud.com
> >>>> > > > > is
> >>>> > > > > > > > dropped
> >>>> > > > > > > >     >> at some point we will have to document the
> > procedure
> >>>> > > > > > > > such
> >>>> > > > that
> >>>> > > > > > > > people know
> >>>> > > > > > > >     >> how to make these stable releases work. That is
> >> where
> >>>> > > > > > > > the
> >>>> > > > > > > procedure
> >>>> > > > > > > >     >> documented here https://shankerbalan.net/blog/
> >>>> > > > > > > >     >> seed-cloudstack-templates-offline/ should be
> >>>> > > > > > > > officially
> >>>> > > > > adapted
> >>>> > > > > > > > with a
> >>>> > > > > > > >     >> marketplace to pick up templates of choosing.
> Maybe
> >>>> > > > > > > > we can
> >>>> > > > > have
> >>>> > > > > > a
> >>>> > > > > > > > pre-setup
> >>>> > > > > > > >     >> script to make choices instead of written wiki.
> > Note
> >>>> > > > > > > > that
> >>>> > > > this
> >>>> > > > > > > will
> >>>> > > > > > > > be
> >>>> > > > > > > >     >> anyway needed by the older releases.
> >>>> > > > > > > >     >>
> >>>> > > > > > > >     >>
> >>>> > > > > > > >     >>
> >>>> > > > > > > >     >>
> >>>> > > > > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
> >>>> > > > paul.angus@shapeblue.com>
> >>>> > > > > > > > wrote:
> >>>> > > > > > > >     >>
> >>>> > > > > > > >     >> >-1
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >We are trying to get MORE people to use
> > CloudStack.
> >>>> > > > > > > > And
> >>>> > > > > > raising
> >>>> > > > > > > > the
> >>>> > > > > > > >     >> barrier to entry does nothing to help that.
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >Kind regards,
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >Paul Angus
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >paul.angus@shapeblue.com
> >>>> > > > > > > >     >> >www.shapeblue.com
> >>>> > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N
> > 4HSUK
> >>>> > > > > > > >     >> >@shapeblue
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >-----Original Message-----
> >>>> > > > > > > >     >> >From: Abhinandan Prateek [mailto:
> >> abhinandan.prateek@
> >>>> > > > > > > shapeblue.com]
> >>>> > > > > > > >     >> >Sent: 01 March 2017 09:39
> >>>> > > > > > > >     >> >To: dev@cloudstack.apache.org
> >>>> > > > > > > >     >> >Subject: Re: Modern template hosting
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >+1
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >After seeing this discussion running in circles
> >>>> > > > > > > > several
> >>>> > > > > times,
> >>>> > > > > > I
> >>>> > > > > > > > think
> >>>> > > > > > > >     >> we should at least get started with the simplest
> >>>> option.
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
> >>>> > > > > > chiradeepv@gmail.com>
> >>>> > > > > > > > wrote:
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >>The alternative foolproof way is to simply not
> >>>> > > > > > > > provide
> >>>> > > the
> >>>> > > > > > > > automatic
> >>>> > > > > > > >     >> >>download.
> >>>> > > > > > > >     >> >>
> >>>> > > > > > > >     >> >>Just document the mirror list URL where the
> >>>> > > > > > > > templates
> >>>> > > can
> >>>> > > > be
> >>>> > > > > > > > found. I
> >>>> > > > > > > >     >> >>don't know why people reject this option.
> >>>> > > > > > > >     >> >>
> >>>> > > > > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
> >>>> > > > > > > templates-offline/
> >>>> > > > > > > >     >> >>
> >>>> > > > > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> >>>> > > > > > > >     >> >><wi...@gmail.com>
> >>>> > > > > > > >     >> >>wrote:
> >>>> > > > > > > >     >> >>
> >>>> > > > > > > >     >> >>> OK. Thanks for the heads up.
> >>>> > > > > > > >     >> >>>
> >>>> > > > > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> >>>> > > > > > > > chiradeepv@gmail.com>
> >>>> > > > > > > >     >> wrote:
> >>>> > > > > > > >     >> >>>
> >>>> > > > > > > >     >> >>> > Sounds workable. The downloader code in the
> >>>> > > > > > > > SSVM
> >>>> > > won't
> >>>> > > > > > > follow
> >>>> > > > > > > >     >> >>> > redirects I think.
> >>>> > > > > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
> >>>> > > > > > > 5511065fc20787619d
> >>>> > > > > > > > 9cd0444
> >>>> > > > > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/
> >>>> > > storage/template/
> >>>> > > > > > > >     >> >>> > HttpTemplateDownloader.java#L93
> >>>> > > > > > > >     >> >>> > https://goo.gl/dSi0r5
> >>>> > > > > > > >     >> >>> >  Might need to add
> >>>> > > > > > > >     >> >>> > client.setRedirectStrategy(new
> >>>> > > LaxRedirectStrategy());
> >>>> > > > > > > >     >> >>> >
> >>>> > > > > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will
> Stevens
> >>>> > > > > > > >     >> >>> > <ws...@cloudops.com>
> >>>> > > > > > > >     >> >>> > wrote:
> >>>> > > > > > > >     >> >>> >
> >>>> > > > > > > >     >> >>> > > We haven't opened a ticket yet because we
> >>>> > > > > > > > don't
> >>>> > > > have a
> >>>> > > > > > > > strategy
> >>>> > > > > > > >     >> yet.
> >>>> > > > > > > >     >> >>> > >
> >>>> > > > > > > >     >> >>> > > What do you guys think of this:
> >>>> > > > > > > >     >> >>> > > - We setup a new github repo in the
> > 'apache'
> >>>> > > > > > > > org
> >>>> > > > which
> >>>> > > > > > > > consists
> >>>> > > > > > > >     >> >>> > > of a
> >>>> > > > > > > >     >> >>> > single
> >>>> > > > > > > >     >> >>> > > file with a list of active/supported
> >> mirrors.
> >>>> > > > > > > >     >> >>> > > - I write a small web server, distributed
> > as
> >>>> > > > > > > > a
> >>>> > > > binary,
> >>>> > > > > > > > which can
> >>>> > > > > > > >     >> >>> > > be
> >>>> > > > > > > >     >> >>> > hosted
> >>>> > > > > > > >     >> >>> > > by ASF Infra.  This web server will query
> >> the
> >>>> > > > current
> >>>> > > > > > list
> >>>> > > > > > > > of
> >>>> > > > > > > >     >> >>> > > mirrors
> >>>> > > > > > > >     >> >>> and
> >>>> > > > > > > >     >> >>> > > will select one and then do a 302
> redirect
> >> to
> >>>> > > > > > > > that
> >>>> > > > > > mirror.
> >>>> > > > > > > >     >> >>> > >
> >>>> > > > > > > >     >> >>> > > The act of 'choosing' a mirror could be
> > done
> >>>> > > > > > > > in a
> >>>> > > > > number
> >>>> > > > > > > of
> >>>> > > > > > > > ways.
> >>>> > > > > > > >     >> >>> > > - If we want to define an order, then it
> >>>> > > > > > > > could
> >>>> > > just
> >>>> > > > > try
> >>>> > > > > > > > from the
> >>>> > > > > > > >     >> >>> > > top of
> >>>> > > > > > > >     >> >>> > the
> >>>> > > > > > > >     >> >>> > > list and work its way down.  It would
> curl
> >>>> > > > > > > > the
> >>>> > > > target
> >>>> > > > > to
> >>>> > > > > > > > make
> >>>> > > > > > > >     >> >>> > > sure it
> >>>> > > > > > > >     >> >>> > gets
> >>>> > > > > > > >     >> >>> > > a 200 and if it does, it would do a 302
> >>>> > redirect.
> >>>> > > > > > > >     >> >>> > > - Or, if we want to distribute the load
> >>>> > > > > > > > across the
> >>>> > > > > > > mirrors,
> >>>> > > > > > > > we
> >>>> > > > > > > >     >> >>> > > could
> >>>> > > > > > > >     >> >>> pick
> >>>> > > > > > > >     >> >>> > > from the list randomly.  Again, doing a
> > curl
> >>>> > > > > > > > to
> >>>> > > > verify
> >>>> > > > > > the
> >>>> > > > > > > > mirror
> >>>> > > > > > > >     >> >>> > > is up
> >>>> > > > > > > >     >> >>> > and
> >>>> > > > > > > >     >> >>> > > then doing a redirect.
> >>>> > > > > > > >     >> >>> > > - If we want to get fancy, we could do a
> >>>> > > > > > > > reverse
> >>>> > > IP
> >>>> > > > > > lookup
> >>>> > > > > > > > and
> >>>> > > > > > > >     >> >>> > > try to
> >>>> > > > > > > >     >> >>> > match
> >>>> > > > > > > >     >> >>> > > the requester with their closest
> >> geographical
> >>>> > > > mirror.
> >>>> > > > > > > >     >> >>> > >
> >>>> > > > > > > >     >> >>> > > Thoughts?
> >>>> > > > > > > >     >> >>> > >
> >>>> > > > > > > >     >> >>> > > *Will STEVENS*
> >>>> > > > > > > >     >> >>> > > Lead Developer
> >>>> > > > > > > >     >> >>> > >
> >>>> > > > > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
> >>>> > > > > > > >     >> >>> > >
> >>>> > > > > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM,
> Chiradeep
> >>>> > > Vittal <
> >>>> > > > > > > >     >> >>> chiradeepv@gmail.com
> >>>> > > > > > > >     >> >>> > >
> >>>> > > > > > > >     >> >>> > > wrote:
> >>>> > > > > > > >     >> >>> > >
> >>>> > > > > > > >     >> >>> > > > What steps are needed to set up a
> mirror?
> >>>> > > > > > > > What
> >>>> > > > does
> >>>> > > > > > > Infra
> >>>> > > > > > > > need
> >>>> > > > > > > >     >> to do?
> >>>> > > > > > > >     >> >>> > Has
> >>>> > > > > > > >     >> >>> > > > anybody filed a ticket with Infra?
> >>>> > > > > > > >     >> >>> > > >
> >>>> > > > > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja
> >>>> > Pullela <
> >>>> > > > > > > >     >> >>> > > > raja.pullela@accelerite.com>
> >>>> > > > > > > >     >> >>> > > > wrote:
> >>>> > > > > > > >     >> >>> > > >
> >>>> > > > > > > >     >> >>> > > > > Hi will,
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > > > I believe, we didn’t get to close
> >>>> > > > > > > > ‘getting a
> >>>> > > > > mirror
> >>>> > > > > > on
> >>>> > > > > > > > Apache’
> >>>> > > > > > > >     >> >>> > because
> >>>> > > > > > > >     >> >>> > > we
> >>>> > > > > > > >     >> >>> > > > > needed someone on the Apache Infra
> side
> >>>> > > > > > > > to
> >>>> > > close
> >>>> > > > > > this.
> >>>> > > > > > > > BTW,
> >>>> > > > > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux
> >>>> > > > > > > > manages
> >>>> > > > this?)
> >>>> > > > > > has
> >>>> > > > > > > >     >> >>> > > > > all/most of
> >>>> > > > > > > >     >> >>> > the
> >>>> > > > > > > >     >> >>> > > > > content.  Once we can close on the
> >> Apache
> >>>> > > mirror
> >>>> > > > > for
> >>>> > > > > > > > hosting
> >>>> > > > > > > >     >> >>> > > > > the
> >>>> > > > > > > >     >> >>> > > > content, I
> >>>> > > > > > > >     >> >>> > > > > can help assist getting the content
> >> there.
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > > > For now, we have replicated the
> >>>> > > > > download.cloud.com
> >>>> > > > > > > > content to
> >>>> > > > > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
> >>>> > > > > > > >     >> >>> > > > > Also, we are working on a set of
> >>>> > > steps/procedure
> >>>> > > > > to
> >>>> > > > > > > > help with
> >>>> > > > > > > >     >> >>> > > > > this change.  I will update everyone
> in
> >>>> > > > > > > > about
> >>>> > > a
> >>>> > > > > > week’s
> >>>> > > > > > > > time
> >>>> > > > > > > >     >> >>> > > > > on the
> >>>> > > > > > > >     >> >>> > details.
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > > > Best,
> >>>> > > > > > > >     >> >>> > > > > Raja Pullela
> >>>> > > > > > > >     >> >>> > > > > Engineering Team,
> >>>> > > > > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road,
> Santa
> >>>> > > > > > > > Clara,
> >>>> > > > CA,
> >>>> > > > > > > 95054
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "
> >>>> > > williamstevens@gmail.com
> >>>> > > > > on
> >>>> > > > > > > > behalf of
> >>>> > > > > > > >     >> >>> > > > > Will Stevens" <
> williamstevens@gmail.com
> >>>> > > > > > > > on
> >>>> > > > behalf
> >>>> > > > > > of
> >>>> > > > > > > >     >> >>> > wstevens@cloudops.com>
> >>>> > > > > > > >     >> >>> > > > > wrote:
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > > > unfortunately the template mirror
> >>>> > > > > > > > conversation
> >>>> > > > got
> >>>> > > > > > > > caught up
> >>>> > > > > > > >     >> >>> > > > > in
> >>>> > > > > > > >     >> >>> > details
> >>>> > > > > > > >     >> >>> > > > and
> >>>> > > > > > > >     >> >>> > > > > nobody took the lead on implementing
> a
> >>>> > > solution.
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > > > citrix has been pinging me every
> couple
> >>>> > > > > > > > months
> >>>> > > > to
> >>>> > > > > > say
> >>>> > > > > > > > 'dude,
> >>>> > > > > > > >     >> >>> > > > > we
> >>>> > > > > > > >     >> >>> need
> >>>> > > > > > > >     >> >>> > to
> >>>> > > > > > > >     >> >>> > > > > remove the dependency on
> >>>> > > > > > > > download.citrix.com
> >>>> > > ',
> >>>> > > > > but
> >>>> > > > > > i
> >>>> > > > > > > > have not
> >>>> > > > > > > >     >> >>> > > > > had
> >>>> > > > > > > >     >> >>> > the
> >>>> > > > > > > >     >> >>> > > > > cycles to get in and solve the
> problem.
> >>>> > > > > > > > the
> >>>> > > > > > shutdown
> >>>> > > > > > > > of that
> >>>> > > > > > > >     >> >>> > > > > is
> >>>> > > > > > > >     >> >>> > > imminent
> >>>> > > > > > > >     >> >>> > > > > right now, so we need to solve it
> asap.
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > > > *Will STEVENS*
> >>>> > > > > > > >     >> >>> > > > > Lead Developer
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM,
> Paul
> >>>> > Angus <
> >>>> > > > > > > >     >> >>> > paul.angus@shapeblue.com
> >>>> > > > > > > >     >> >>> > > >
> >>>> > > > > > > >     >> >>> > > > > wrote:
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > > > > Hi Nathan,
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > > Ideally, if you put the template
> >>>> > > > > > > > location in
> >>>> > > > (or
> >>>> > > > > > use
> >>>> > > > > > > a
> >>>> > > > > > > >     >> >>> > > > > > template
> >>>> > > > > > > >     >> >>> > > defined
> >>>> > > > > > > >     >> >>> > > > > > in)  test_data.py then the actual
> >>>> > > > > > > > location
> >>>> > > can
> >>>> > > > > be
> >>>> > > > > > > >     >> >>> > > > > > overridden by
> >>>> > > > > > > >     >> >>> > > anyone
> >>>> > > > > > > >     >> >>> > > > > > testing.
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > > For Trillian, we've copied all of
> the
> >>>> > > > templates
> >>>> > > > > > that
> >>>> > > > > > > > people
> >>>> > > > > > > >     >> >>> > > > > > have
> >>>> > > > > > > >     >> >>> > > define
> >>>> > > > > > > >     >> >>> > > > > to
> >>>> > > > > > > >     >> >>> > > > > > a local repo and then replace the
> > URLs
> >>>> > > > > > > > in test_data.py to
> >>>> > > > > > > >     >> >>> > > > > > reduce
> >>>> > > > > > > >     >> >>> > > > > bandwidth
> >>>> > > > > > > >     >> >>> > > > > > use and download times.
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > > Ie:
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > >             "bootableIso":
> >>>> > > > > > > >     >> >>> > > > > >                 {
> >>>> > > > > > > >     >> >>> > > > > >                     "displaytext":
> >> "Test
> >>>> > > > > Bootable
> >>>> > > > > > > > ISO",
> >>>> > > > > > > >     >> >>> > > > > >                     "name":
> > "testISO",
> >>>> > > > > > > >     >> >>> > > > > >                     "bootable":
> True,
> >>>> > > > > > > >     >> >>> > > > > >                     "ispublic":
> > False,
> >>>> > > > > > > >     >> >>> > > > > >                     "url": "{{
> >>>> > > > > > > marvin_images_location
> >>>> > > > > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
> >>>> > > > > > > >     >> >>> > > > > >                     "ostype":
> 'Other
> >>>> Linux
> >>>> > > > > > > (64-bit)',
> >>>> > > > > > > >     >> >>> > > > > >                     "mode":
> >>>> > 'HTTP_DOWNLOAD'
> >>>> > > > > > > >     >> >>> > > > > >         },
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > > I thought that we had come up with
> a
> >>>> > > solution
> >>>> > > > > for
> >>>> > > > > > > >     >> >>> > download.cloud.com
> >>>> > > > > > > >     >> >>> > > ,
> >>>> > > > > > > >     >> >>> > > > by
> >>>> > > > > > > >     >> >>> > > > > > having a mirrorlist hosted in
> >> Community
> >>>> > > Apache
> >>>> > > > > > > > 'space' with
> >>>> > > > > > > >     >> >>> anyone
> >>>> > > > > > > >     >> >>> > > able
> >>>> > > > > > > >     >> >>> > > > > to
> >>>> > > > > > > >     >> >>> > > > > > out themselves forward as a mirror.
> >>>> > > > > > > >     >> >>> > > > > > But I must admit I lost track of
> >>>> > > > > > > > whether
> >>>> > > > anyone
> >>>> > > > > > made
> >>>> > > > > > > > the
> >>>> > > > > > > >     >> >>> requisite
> >>>> > > > > > > >     >> >>> > > > > changes
> >>>> > > > > > > >     >> >>> > > > > > in code....
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > > Kind regards,
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > > Paul Angus
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > > paul.angus@shapeblue.com
> >>>> > > > > > > >     >> >>> > > > > > www.shapeblue.com
> >>>> > > > > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden,
> >> London
> >>>> > > WC2N
> >>>> > > > > > 4HSUK
> >>>> > > > > > > >     >> >>> > > > > > @shapeblue
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > > -----Original Message-----
> >>>> > > > > > > >     >> >>> > > > > > From: williamstevens@gmail.com
> >>>> > > > > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
> >>>> > > > > > > >     >> >>> > On
> >>>> > > > > > > >     >> >>> > > > > > Behalf Of Will Stevens
> >>>> > > > > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
> >>>> > > > > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
> >>>> > > > > > > >     >> >>> > > > > > Subject: Re: Modern template
> hosting
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > > this is a hard questions.  in
> > general,
> >>>> > > > > > > > we
> >>>> > > > should
> >>>> > > > > > be
> >>>> > > > > > > > setting
> >>>> > > > > > > >     >> >>> > > > > > up a
> >>>> > > > > > > >     >> >>> > > mirror
> >>>> > > > > > > >     >> >>> > > > > on
> >>>> > > > > > > >     >> >>> > > > > > some cloudstack/apache domain and
> > then
> >>>> > > mirror
> >>>> > > > to
> >>>> > > > > > > other
> >>>> > > > > > > >     >> >>> > > > > > provided
> >>>> > > > > > > >     >> >>> > > > > templates.
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > > we MUST come up with a solution to
> >>>> > deprecate
> >>>> > > > > > > >     >> >>> > > > > > 'download.cloud.com
> >>>> > > > > > > >     >> >>> ',
> >>>> > > > > > > >     >> >>> > > > that
> >>>> > > > > > > >     >> >>> > > > > > is going to be going away any day
> > now.
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > > i don't know the right way to solve
> >>>> > > > > > > > this to
> >>>> > > be
> >>>> > > > > > > > honest, but
> >>>> > > > > > > >     >> >>> > > > > > if you
> >>>> > > > > > > >     >> >>> > > have
> >>>> > > > > > > >     >> >>> > > > > > ideas, i am willing to help.
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > > *Will STEVENS*
> >>>> > > > > > > >     >> >>> > > > > > Lead Developer
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM,
> >>>> > > > > > > > Nathan
> >>>> > > > > Johnson <
> >>>> > > > > > > >     >> >>> njohnson@ena.com
> >>>> > > > > > > >     >> >>> > >
> >>>> > > > > > > >     >> >>> > > > > wrote:
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > > > > So not to re-open a can of worms,
> >> but
> >>>> > > > > > > > I’m
> >>>> > > > in a
> >>>> > > > > > > > situation
> >>>> > > > > > > >     >> >>> > > > > > > where
> >>>> > > > > > > >     >> >>> I
> >>>> > > > > > > >     >> >>> > > need
> >>>> > > > > > > >     >> >>> > > > > > > to come up with a Marvin
> component
> >>>> > > > > > > > test
> >>>> > > that
> >>>> > > > > > > > depends on a
> >>>> > > > > > > >     >> >>> > template
> >>>> > > > > > > >     >> >>> > > > > > > based on a kernel that’s
> relatively
> >>>> > > > > > > > new,
> >>>> > > > i.e.,
> >>>> > > > > > > > newer than
> >>>> > > > > > > >     >> >>> Centos
> >>>> > > > > > > >     >> >>> > > 5.3
> >>>> > > > > > > >     >> >>> > > > /
> >>>> > > > > > > >     >> >>> > > > > > Ubuntu 10.04 .
> >>>> > > > > > > >     >> >>> > > > > > > I see openvm.eu has a suitable
> >>>> > > > > > > > template
> >>>> > > > > (Ubuntu
> >>>> > > > > > > > 16.0.4
> >>>> > > > > > > >     >> >>> > > > > > > for
> >>>> > > > > > > >     >> >>> KVM),
> >>>> > > > > > > >     >> >>> > > but
> >>>> > > > > > > >     >> >>> > > > > > > from looking at the thread
> >> "Migrating
> >>>> > > > > CloudStack
> >>>> > > > > > > > content
> >>>> > > > > > > >     >> >>> > > > > > > from download.cloud.com” it
> looks
> >>>> > > > > > > > like
> >>>> > > > there
> >>>> > > > > is
> >>>> > > > > > > >     >> >>> > > > > > > resistance to using
> >>>> > > > > > > >     >> >>> > > this
> >>>> > > > > > > >     >> >>> > > > at
> >>>> > > > > > > >     >> >>> > > > > > > least for hosting system vm
> >> templates
> >>>> > > > > > > > over
> >>>> > > > > > > concerns
> >>>> > > > > > > > of
> >>>> > > > > > > >     >> >>> > neutrality.
> >>>> > > > > > > >     >> >>> > > > > > > Would this be suitable for a
> >>>> > > > > > > > component
> >>>> > > test?
> >>>> > > > > If
> >>>> > > > > > > > not,
> >>>> > > > > > > >     >> >>> > > > > > > what is a
> >>>> > > > > > > >     >> >>> > > > > “blessed”
> >>>> > > > > > > >     >> >>> > > > > > > template location?
> >>>> > > > > > > >     >> >>> > > > > > >
> >>>> > > > > > > >     >> >>> > > > > > > Thanks in advance!
> >>>> > > > > > > >     >> >>> > > > > > >
> >>>> > > > > > > >     >> >>> > > > > > > Nathan Johnson
> >>>> > > > > > > >     >> >>> > > > > > > R&D Engineer
> >>>> > > > > > > >     >> >>> > > > > > >
> >>>> > > > > > > >     >> >>> > > > > > >
> >>>> > > > > > > >     >> >>> > > > > > >
> >>>> > > > > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite
> 12
> >>>> > > > Nashville,
> >>>> > > > > TN
> >>>> > > > > > > > 37211
> >>>> > > > > > > >     >> >>> > > > > > > General Office: 615-312-6000
> >>>> > > > > > > >     >> >>> > > > > > >
> >>>> > > > > > > >     >> >>> > > > > > > website | blog | support
> >>>> > > > > > > >     >> >>> > > > > > >
> >>>> > > > > > > >     >> >>> > > > > > >
> >>>> > > > > > > >     >> >>> > > > > > >
> >>>> > > > > > > >     >> >>> > > > > > >
> >>>> > > > > > > >     >> >>> > > > > > >
> >>>> > > > > > > >     >> >>> > > > > > >
> >>>> > > > > > > >     >> >>> > > > > > >
> >>>> > > > > > > >     >> >>> > > > > >
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > > > DISCLAIMER
> >>>> > > > > > > >     >> >>> > > > > ==========
> >>>> > > > > > > >     >> >>> > > > > This e-mail may contain privileged
> and
> >>>> > > > > confidential
> >>>> > > > > > > >     >> >>> > > > > information
> >>>> > > > > > > >     >> >>> which
> >>>> > > > > > > >     >> >>> > > is
> >>>> > > > > > > >     >> >>> > > > > the property of Accelerite, a
> > Persistent
> >>>> > > Systems
> >>>> > > > > > > > business. It
> >>>> > > > > > > >     >> >>> > > > > is
> >>>> > > > > > > >     >> >>> > > intended
> >>>> > > > > > > >     >> >>> > > > > only for the use of the individual or
> >>>> > > > > > > > entity
> >>>> > > to
> >>>> > > > > > which
> >>>> > > > > > > > it is
> >>>> > > > > > > >     >> >>> > addressed.
> >>>> > > > > > > >     >> >>> > > If
> >>>> > > > > > > >     >> >>> > > > > you are not the intended recipient,
> you
> >>>> > > > > > > > are
> >>>> > > not
> >>>> > > > > > > > authorized to
> >>>> > > > > > > >     >> >>> > > > > read,
> >>>> > > > > > > >     >> >>> > > > retain,
> >>>> > > > > > > >     >> >>> > > > > copy, print, distribute or use this
> >>>> message.
> >>>> > > If
> >>>> > > > > you
> >>>> > > > > > > have
> >>>> > > > > > > >     >> >>> > > > > received
> >>>> > > > > > > >     >> >>> > this
> >>>> > > > > > > >     >> >>> > > > > communication in error, please notify
> >> the
> >>>> > > sender
> >>>> > > > > and
> >>>> > > > > > > > delete
> >>>> > > > > > > >     >> >>> > > > > all
> >>>> > > > > > > >     >> >>> > copies
> >>>> > > > > > > >     >> >>> > > of
> >>>> > > > > > > >     >> >>> > > > > this message. Accelerite, a
> Persistent
> >>>> > > > > > > > Systems
> >>>> > > > > > > business
> >>>> > > > > > > > does
> >>>> > > > > > > >     >> >>> > > > > not
> >>>> > > > > > > >     >> >>> > accept
> >>>> > > > > > > >     >> >>> > > > any
> >>>> > > > > > > >     >> >>> > > > > liability for virus infected mails.
> >>>> > > > > > > >     >> >>> > > > >
> >>>> > > > > > > >     >> >>> > > >
> >>>> > > > > > > >     >> >>> > >
> >>>> > > > > > > >     >> >>> >
> >>>> > > > > > > >     >> >>>
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >abhinandan.prateek@shapeblue.com
> >>>> > > > > > > >     >> >www.shapeblue.com
> >>>> > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N
> > 4HSUK
> >>>> > > > > @shapeblue
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >> >
> >>>> > > > > > > >     >>
> >>>> > > > > > > >     >> abhinandan.prateek@shapeblue.com
> >>>> > > > > > > >     >> www.shapeblue.com
> >>>> > > > > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N
> 4HSUK
> >>>> > > > > > > >     >> @shapeblue
> >>>> > > > > > > >     >>
> >>>> > > > > > > >     >>
> >>>> > > > > > > >     >>
> >>>> > > > > > > >     >>
> >>>> > > > > > > >     >
> >>>> > > > > > > >
> >>>> > > > > > > >
> >>>> > > > > > > >
> >>>> > > > > > > > daan.hoogland@shapeblue.com
> >>>> > > > > > > > www.shapeblue.com
> >>>> > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >>>> > > > > > > > @shapeblue
> >>>> > > > > > > >
> >>>> > > > > > > >
> >>>> > > > > > > >
> >>>> > > > > > > >
> >>>> > > > > > >
> >>>> > > > > > >
> >>>> > > > > > > --
> >>>> > > > > > > Rafael Weingärtner
> >>>> > > > > > >
> >>>> > > > > >
> >>>> > > > > >
> >>>> > > > > >
> >>>> > > > > > --
> >>>> > > > > > Rafael Weingärtner
> >>>> > > > > >
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
>
>

Re: Modern template hosting

Posted by Erik Weber <te...@gmail.com>.

Feel free to do it, I wouldn't know where to begin to ask such a
question tbh :-)

-- 
Erik

On Sat, Mar 4, 2017 at 12:51 AM, Will Stevens <wi...@gmail.com> wrote:
> Erik, yes I think this is the right approach and covers the main problem
> case. I can start the conversation with infra unless you would like to. Let
> me know.
>
> We did get some statistics from the other mirrors last year, but I don't
> think we ever had visibility into the cloud.com repository traffic.
>
> On Mar 3, 2017 6:42 PM, "Erik Weber" <te...@gmail.com> wrote:
>
> It'll most likely only be an issue in the case where a mirror goes
> down without any immediate chance of getting back up, right?
> Could we check with ASF Infra beforehand if such case is OK to warrant
> an urgent Jira ticket to resolve - should it happen?
>
> We can document all the mirrors and provide the howtos for manually
> downloading and seeding the image as a backup.
>
> Do we have any traffic statistics for the current solution, or is that
> not available at S3?
>
> --
> Erik
>
> On Sat, Mar 4, 2017 at 12:28 AM, Will Stevens <wi...@gmail.com>
> wrote:
>> Yes Erik, I agree with you. The only thing that could add complexity in
>> this is the fact that we don't control the domain, the ASF does (from what
>> I understand).  Do we expect this to be managed by ASF infra or would
> there
>> be another way? Ideally the domain is community controlled so we can adapt
>> in case of change without long delays.
>>
>> I am 100% in agreement that this is the shortest path and likely to be
> good
>> enough.
>>
>> On Mar 3, 2017 6:19 PM, "Erik Weber" <te...@gmail.com> wrote:
>>
>> IMHO; implement it as simple as possible, use DNS RR and assume/expect
>> mirror hosts to use the same path.
>>
>>
>> Yes I know; DNS RR isn't bulletproof, but as already mentioned in this
>> thread - mirrors rarely go down and they aren't used all that often
>> (primarily seeding).
>> We can keep a low TTL on the record so that we're able to remove a
>> mirror that goes down for a significant period of time (more than
>> hours).
>>
>> And yes I know, some operators might find it a burden having to set up
>> a new vhost with another document root, but tbh if they don't care
>> enough to do it we'll manage without that mirror.
>>
>> --
>> Erik
>>
>> On Sat, Mar 4, 2017 at 12:09 AM, Will Stevens <wi...@gmail.com>
>> wrote:
>>> I agree with Paul. Look at the list of things they have to
>>> learn/master/care about. We don't want to add to that list, we want to
>>> remove from that list. Make the project/product more accessible. Reduce
>> the
>>> barrier to entry...
>>>
>>> On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:
>>>
>>>> Seriously?
>>>> apt-get install apache2
>>>>
>>>> To install CloudStack, they need to know
>>>> - DB installation, perhaps some SQL
>>>> - VLAN configuration on their switches
>>>> - Ins-and-outs between port forwarding, static ips,
>>>> - NFS
>>>> - package management
>>>> - VHDs, qcow2. vmdk
>>>> - hyperviosrs
>>>> - and on and on.
>>>>
>>>> If they can figure all that out and not figure out a web server, they
> can
>>>> always come to the mailing list.
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <pa...@shapeblue.com>
>>>> wrote:
>>>>
>>>> > The issue is not with supporting highly experienced cloud operators.
> It
>>>> is
>>>> > an issue for new users and other 'relatively' inexperienced operators.
>>>> >
>>>> > I have helped enough newbies and cloud operators who have been running
>>>> > their cloud for a while to know that the barriers to entry are too
> high
>>>> as
>>>> > they are. And telling anyone that they need to create a web server so
>>>> that
>>>> > they can add their initial template to get started or in order to
>> create
>>>> a
>>>> > new zone, just isn't going to fly.
>>>> >
>>>> > I'm a huge advocate of the 'download.cloudstack.org' endpoint which
> the
>>>> > community can add/remove mirrors to or from for system VMs or built-in
>>>> > templates.  Can the same system be used for binary repos ? although
>> there
>>>> > is an added complication of redist vs no-redist there...
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > paul.angus@shapeblue.com
>>>> > www.shapeblue.com
>>>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>>> > @shapeblue
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > -----Original Message-----
>>>> > From: Chiradeep Vittal [mailto:chiradeepv@gmail.com]
>>>> > Sent: 03 March 2017 18:28
>>>> > To: dev <de...@cloudstack.apache.org>
>>>> > Subject: Re: Modern template hosting
>>>> >
>>>> > I do feel like this is early optimization. Mirrors rarely fail. I'd
>>>> expect
>>>> > a single web server hosted on Apache Infra without any monitors to
> fail
>>>> > more often than a mirror. We already expect Wido's systemvm repository
>> to
>>>> > be up all the time. And it has been. Similarly, I don't believe Nux's
>>>> > repository has ever been down. And if Accelerite wants to host on S3,
>>>> that
>>>> > one is pretty solid as well.
>>>> >
>>>> > This is an infrequent operation in a cloud. After the cloud is
>> installed,
>>>> > the download servers are only needed for a new zone. If we trust the
>> user
>>>> > to run a cloud, surely he/she can run a web server to serve some
>> built-in
>>>> > templates. And if her cloud is successful, she needs to figure out how
>> to
>>>> > host her templates anyway and not rely on 3rd parties.
>>>> >
>>>> >
>>>> > On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <ws...@cloudops.com>
>>>> > wrote:
>>>> >
>>>> > > 1) If the legacy implementations do not support redirects, that does
>>>> > > cause a problem.  A potential solution in that case is to have the
>> web
>>>> > > server actually proxy the download, but that is not ideal and I
> would
>>>> > > like to avoid it if possible.  Thanks for bringing that up
> Chiradeep.
>>>> > >
>>>> > > 2) I think we need to have a single URL which people can target.
>> Once
>>>> > > they make the switch to the new URL, we want the implementation to
> be
>>>> > > able to handle mirror failures without affecting the end client.  We
>>>> > > want to avoid the situation where an ACS user will ever have to
>> change
>>>> > > this URL more than once.  Mirror failures SHOULD NOT affect the ACS
>>>> > > users assuming there is still at least one mirror who can serve the
>>>> > requested resource.
>>>> > >
>>>> > > These are obviously my personal opinions and others will probably
>> have
>>>> > > differing opinions.
>>>> > >
>>>> > > *Will STEVENS*
>>>> > > Lead Developer
>>>> > >
>>>> > > <https://goo.gl/NYZ8KK>
>>>> > >
>>>> > > On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal
>>>> > > <ch...@gmail.com>
>>>> > > wrote:
>>>> > >
>>>> > > > 1. If you are targeting legacy installations, they are not able to
>>>> > > > follow redirects. The line of code that added this capability was
>>>> > > > added on
>>>> > > 11/16.
>>>> > > > 2. If you trust the users to edit the database to change the URL,
>>>> > > > you can trust them to change it to anything. Just document a known
>>>> > > > list of good template locations.
>>>> > > >
>>>> > > > On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <
> wstevens@cloudops.com
>>>
>>>> > > > wrote:
>>>> > > >
>>>> > > > > So the main issue I see with this is the following.
>>>> > > > >
>>>> > > > > This implementation is designed to target legacy installations
>>>> > > > > which
>>>> > > will
>>>> > > > > be affected when download.cloud.com disappears.  These people
>> will
>>>> > > need
>>>> > > > to
>>>> > > > > make a DB change to replace the 'download.cloud.com' with some
>>>> > > > > other
>>>> > > url
>>>> > > > > (maybe 'download.cloudstack.org').  Once the DB has been
> updated,
>>>> > > > > we
>>>> > > can
>>>> > > > > not expect anything else of the client, they need to be able to
>>>> > > > > just continue operation as they were without any need to rebuild
>>>> > > > > or upgrade their ACS.
>>>> > > > >
>>>> > > > > If we try to force the decision to the client, then we break the
>>>> > > > > legacy implementations.  Since the SSVM is likely going to be
> the
>>>> > > > > client in
>>>> > > some
>>>> > > > > cases, and since it does not already have the logic to handle
> the
>>>> > > > > 300 approach correctly, I feel like it is not a viable solution
>>>> > > > > for the
>>>> > > > legacy
>>>> > > > > deployments.
>>>> > > > >
>>>> > > > > In addition to that.  I don't think the client is in any better
>>>> > > position
>>>> > > > to
>>>> > > > > make the mirror decision than I am.  I am likely able to make a
>>>> > > > > more 'educated' decision than the client would be able to
> because
>>>> > > > > I can do different tests on the endpoint(s) before making a
> final
>>>> > decision.
>>>> > > > >
>>>> > > > > Does this make sense or am I missing something obvious?
>>>> > > > >
>>>> > > > > *Will STEVENS*
>>>> > > > > Lead Developer
>>>> > > > >
>>>> > > > > <https://goo.gl/NYZ8KK>
>>>> > > > >
>>>> > > > > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
>>>> > > > > rafaelweingartner@gmail.com> wrote:
>>>> > > > >
>>>> > > > > > I think I understood you, but I did not understand you.
>>>> > > > > >
>>>> > > > > > Let me see if I can create an example to illustrate how we
>> could
>>>> > > > > > do
>>>> > > > using
>>>> > > > > > the HTTP 300 code.
>>>> > > > > >
>>>> > > > > > The HTTP 300 code indicates clients that he/she/it has
> multiple
>>>> > > choices
>>>> > > > > to
>>>> > > > > > access the requested resource. If the request is a HEAD, this
>> is
>>>> > > > > > the
>>>> > > > only
>>>> > > > > > thing that the server returns. If the request if a GET, then
>> the
>>>> > > server
>>>> > > > > > returns the choices for the client. The response type is
>> defined
>>>> > > > > > on
>>>> > > the
>>>> > > > > > server based on the Content-Type, user agent and others.
>>>> > > > > >
>>>> > > > > > Let’s say the server receives a GET request and informs the
>>>> > > > content-type
>>>> > > > > as
>>>> > > > > > JSON (application/JSON). Your application could get the
> mirrors
>>>> > > > > > list,
>>>> > > > do
>>>> > > > > > the appending process for the requested resource path, and
> then
>>>> > > return
>>>> > > > a
>>>> > > > > > JSON array with the possible mirror the client can use. Then,
>>>> > > > > > the
>>>> > > > client
>>>> > > > > > gets this list and does the processing required to select a
>>>> mirror.
>>>> > > > > >
>>>> > > > > > Now let’s say that a human access the link using a browser.
>>>> > > > > > Then, the server should receive something like (text/plain) as
>>>> > > > > > the
>>>> > > content-type;
>>>> > > > I
>>>> > > > > > think depending on the browser this field may differ a little
>>>> > > > > > (it
>>>> > > would
>>>> > > > > > require some checking). Anyways, the server detects that it is
>> a
>>>> > > > “human”
>>>> > > > > > requesting the resource, then we could serve an HTML page with
>> a
>>>> > > > > Javascript
>>>> > > > > > that uses the mirror list. This Javascript could do some
>>>> > > > > > checking and choose the best mirror for that specific.
>>>> > > > > >
>>>> > > > > > Why do I think this approach is interesting?
>>>> > > > > > The application you developed would be used only to retrieve
>>>> > > > > > valid
>>>> > > and
>>>> > > > > > trusted sources of system VM images (a trusted repository for
>>>> > > > > > mirrors
>>>> > > > > that
>>>> > > > > > we as a community vouch for).  Also, we shift the decision
>>>> > > > > > process regarding mirrors from the server to the client. Then,
>>>> > > > > > it is up to
>>>> > > > > clients
>>>> > > > > > to select mirrors, and not up to us (or some of our
>>>> > implementations).
>>>> > > > > >
>>>> > > > > > Did this help?
>>>> > > > > > BTW: I am not saying I am against the way you proposed, which
>>>> > > > > > would
>>>> > > > work
>>>> > > > > > fine. It is merely a suggestion using a different perspective.
>>>> > > > > >
>>>> > > > > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
>>>> > > > williamstevens@gmail.com>
>>>> > > > > > wrote:
>>>> > > > > >
>>>> > > > > > > @rafael: in general, I don't think the client should ever be
>>>> > > making a
>>>> > > > > > > choice about a mirror. We have to assume we are working with
>> a
>>>> > > > scripted
>>>> > > > > > > application and anything hard coded in that implementation
> on
>>>> > > > > > > the
>>>> > > > > client
>>>> > > > > > > side is a risk.
>>>> > > > > > >
>>>> > > > > > > I may not be understanding the use of the 300 approach, so I
>>>> > > > > > > will
>>>> > > > > > research
>>>> > > > > > > it to see if I can make it fit.
>>>> > > > > > >
>>>> > > > > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
>>>> > > > > > rafaelweingartner@gmail.com>
>>>> > > > > > > wrote:
>>>> > > > > > >
>>>> > > > > > > Will great job.
>>>> > > > > > >
>>>> > > > > > > I had the same doubt as Daan.
>>>> > > > > > >
>>>> > > > > > > Have you considered using HTTP 300 working mode? Then, we
>>>> > > > > > > could let
>>>> > > > the
>>>> > > > > > > client decide which mirror is the best(closest?). Section
>>>> > “10.3.1”
>>>> > > of
>>>> > > > > [1]
>>>> > > > > > > talks about it; this would put the pressure on deciding upon
>> a
>>>> > > mirror
>>>> > > > > on
>>>> > > > > > > the client side, but I think it would be fair.
>>>> > > > > > >
>>>> > > > > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
>>>> > > > > > >
>>>> > > > > > > PS: You really like very short variables names!
>>>> > > > > > >
>>>> > > > > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
>>>> > > > > > daan.hoogland@shapeblue.com
>>>> > > > > > > >
>>>> > > > > > > wrote:
>>>> > > > > > >
>>>> > > > > > > > Nice little thing Will,
>>>> > > > > > > >
>>>> > > > > > > > One question: if I read the code correctly it ‘go’es and
>>>> > > > > > > > tries
>>>> > > all
>>>> > > > > > > mirrors
>>>> > > > > > > > at once and whichever responses first is redirected to the
>>>> > > client.
>>>> > > > > This
>>>> > > > > > > > might well be the same every time. This might be the one
>>>> > > > > > > > closest
>>>> > > to
>>>> > > > > the
>>>> > > > > > > > mirror302 and not to the client. Is that correct or did I
>>>> > > > > > > > miss a
>>>> > > > > > weighing
>>>> > > > > > > > algorithm hidden in there?
>>>> > > > > > > >
>>>> > > > > > > > Good coding,
>>>> > > > > > > >
>>>> > > > > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org>
> wrote:
>>>> > > > > > > >
>>>> > > > > > > >     Hey All,
>>>> > > > > > > >     Please review this repo and tell me what you think:
>>>> > > > > > > >     https://github.com/swill/mirror302
>>>> > > > > > > >
>>>> > > > > > > >     Let me know if you have questions or you would like me
>>>> > > > > > > > to
>>>> > > make
>>>> > > > > > > changes
>>>> > > > > > > >     before I bring this topic up with ASF Infra and open a
>>>> > > > > > > > ticket
>>>> > > > to
>>>> > > > > > get
>>>> > > > > > > > this
>>>> > > > > > > >     implemented.
>>>> > > > > > > >
>>>> > > > > > > >     Cheers,
>>>> > > > > > > >
>>>> > > > > > > >     *Will Stevens*
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
>>>> > > > > > wstevens@cloudops.com
>>>> > > > > > > >
>>>> > > > > > > > wrote:
>>>> > > > > > > >
>>>> > > > > > > >     > I am building a short term solution right now.  I
>>>> > > > > > > > hopefully
>>>> > > > > will
>>>> > > > > > > > have the
>>>> > > > > > > >     > building blocks in place by the weekend so we can
>>>> > > > > > > > start
>>>> > > > working
>>>> > > > > > > with
>>>> > > > > > > > Infra
>>>> > > > > > > >     > to get it in place.
>>>> > > > > > > >     >
>>>> > > > > > > >     > We will have a web server which we can point
>> something
>>>> > like
>>>> > > > > > > >     > downloads.cloudstack.org (or whatever url) at.  Then
>>>> > > > > > > > we
>>>> > > will
>>>> > > > > > have
>>>> > > > > > > > an ASF
>>>> > > > > > > >     > repo which tracks a mirror list and exposes a static
>>>> > > > > > > > site
>>>> > > to
>>>> > > > be
>>>> > > > > > > able
>>>> > > > > > > > to
>>>> > > > > > > >     > query the mirror list.
>>>> > > > > > > >     >
>>>> > > > > > > >     > The web server will receive a request and will do a
>>>> > > > > > > > 302
>>>> > > > > redirect
>>>> > > > > > to
>>>> > > > > > > > the
>>>> > > > > > > >     > appropriate resource on one of the mirrors.
>>>> > > > > > > >     >
>>>> > > > > > > >     > This gets us started.  Legacy environments will have
>>>> > > > > > > > to do
>>>> > > a
>>>> > > > DB
>>>> > > > > > > > change to
>>>> > > > > > > >     > change from 'download.cloud.com' to '
>>>> > > > downloads.cloudstack.org'
>>>> > > > > > (or
>>>> > > > > > > >     > whatever), but otherwise it should be pretty simple.
>>>> > > > > > > >     >
>>>> > > > > > > >     > We can improve how we deliver templates going
>> forward,
>>>> > > > > > > > but
>>>> > > > this
>>>> > > > > > > > obviously
>>>> > > > > > > >     > requires some discussion still.  I will try to buy
> us
>>>> > > > > > > > some
>>>> > > > time
>>>> > > > > > > with
>>>> > > > > > > > an
>>>> > > > > > > >     > implementation which solves our problems today...
>>>> > > > > > > >     >
>>>> > > > > > > >     > *Will STEVENS*
>>>> > > > > > > >     > Lead Developer
>>>> > > > > > > >     >
>>>> > > > > > > >     > <https://goo.gl/NYZ8KK>
>>>> > > > > > > >     >
>>>> > > > > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
>>>> > > > > > > >     > abhinandan.prateek@shapeblue.com> wrote:
>>>> > > > > > > >     >
>>>> > > > > > > >     >>
>>>> > > > > > > >     >> Initial seeding is a manual step and that is not
>>>> > > > > > > > going to
>>>> > > > > change
>>>> > > > > > > in
>>>> > > > > > > > near
>>>> > > > > > > >     >> future. A handy list of official places from where
>>>> > > > > > > > these
>>>> > > > > > templates
>>>> > > > > > > > can be
>>>> > > > > > > >     >> downloaded won’t harm this part of the
> installation.
>>>> > > > > > > > Just
>>>> > > > > allow
>>>> > > > > > > > anyone
>>>> > > > > > > >     >> credible who follows democratic process to publish
>>>> > > > > > > > their
>>>> > > > > > templates
>>>> > > > > > > > with
>>>> > > > > > > >     >> apache’s blessings.
>>>> > > > > > > >     >>
>>>> > > > > > > >     >>
>>>> > > > > > > >     >> Coming to urls hard coded in db, the popular
>>>> > > > > > > > suggestion is
>>>> > > > to
>>>> > > > > > > point
>>>> > > > > > > > them
>>>> > > > > > > >     >> to mirrors. This looks good but this is going to
>> take
>>>> > > > > > > > some
>>>> > > > > > effort
>>>> > > > > > > > that has
>>>> > > > > > > >     >> not materialised yet. Once this change is made it
>>>> > > > > > > > will be
>>>> > > > > going
>>>> > > > > > in
>>>> > > > > > > > some
>>>> > > > > > > >     >> future release like 4.11; the problem still remains
>>>> > > > > > > > for
>>>> > > the
>>>> > > > > > > > releases that
>>>> > > > > > > >     >> are already out there. In case access to
>>>> > > download.cloud.com
>>>> > > > > is
>>>> > > > > > > > dropped
>>>> > > > > > > >     >> at some point we will have to document the
> procedure
>>>> > > > > > > > such
>>>> > > > that
>>>> > > > > > > > people know
>>>> > > > > > > >     >> how to make these stable releases work. That is
>> where
>>>> > > > > > > > the
>>>> > > > > > > procedure
>>>> > > > > > > >     >> documented here https://shankerbalan.net/blog/
>>>> > > > > > > >     >> seed-cloudstack-templates-offline/ should be
>>>> > > > > > > > officially
>>>> > > > > adapted
>>>> > > > > > > > with a
>>>> > > > > > > >     >> marketplace to pick up templates of choosing. Maybe
>>>> > > > > > > > we can
>>>> > > > > have
>>>> > > > > > a
>>>> > > > > > > > pre-setup
>>>> > > > > > > >     >> script to make choices instead of written wiki.
> Note
>>>> > > > > > > > that
>>>> > > > this
>>>> > > > > > > will
>>>> > > > > > > > be
>>>> > > > > > > >     >> anyway needed by the older releases.
>>>> > > > > > > >     >>
>>>> > > > > > > >     >>
>>>> > > > > > > >     >>
>>>> > > > > > > >     >>
>>>> > > > > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
>>>> > > > paul.angus@shapeblue.com>
>>>> > > > > > > > wrote:
>>>> > > > > > > >     >>
>>>> > > > > > > >     >> >-1
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >We are trying to get MORE people to use
> CloudStack.
>>>> > > > > > > > And
>>>> > > > > > raising
>>>> > > > > > > > the
>>>> > > > > > > >     >> barrier to entry does nothing to help that.
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >Kind regards,
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >Paul Angus
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >paul.angus@shapeblue.com
>>>> > > > > > > >     >> >www.shapeblue.com
>>>> > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N
> 4HSUK
>>>> > > > > > > >     >> >@shapeblue
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >-----Original Message-----
>>>> > > > > > > >     >> >From: Abhinandan Prateek [mailto:
>> abhinandan.prateek@
>>>> > > > > > > shapeblue.com]
>>>> > > > > > > >     >> >Sent: 01 March 2017 09:39
>>>> > > > > > > >     >> >To: dev@cloudstack.apache.org
>>>> > > > > > > >     >> >Subject: Re: Modern template hosting
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >+1
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >After seeing this discussion running in circles
>>>> > > > > > > > several
>>>> > > > > times,
>>>> > > > > > I
>>>> > > > > > > > think
>>>> > > > > > > >     >> we should at least get started with the simplest
>>>> option.
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
>>>> > > > > > chiradeepv@gmail.com>
>>>> > > > > > > > wrote:
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >>The alternative foolproof way is to simply not
>>>> > > > > > > > provide
>>>> > > the
>>>> > > > > > > > automatic
>>>> > > > > > > >     >> >>download.
>>>> > > > > > > >     >> >>
>>>> > > > > > > >     >> >>Just document the mirror list URL where the
>>>> > > > > > > > templates
>>>> > > can
>>>> > > > be
>>>> > > > > > > > found. I
>>>> > > > > > > >     >> >>don't know why people reject this option.
>>>> > > > > > > >     >> >>
>>>> > > > > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
>>>> > > > > > > templates-offline/
>>>> > > > > > > >     >> >>
>>>> > > > > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
>>>> > > > > > > >     >> >><wi...@gmail.com>
>>>> > > > > > > >     >> >>wrote:
>>>> > > > > > > >     >> >>
>>>> > > > > > > >     >> >>> OK. Thanks for the heads up.
>>>> > > > > > > >     >> >>>
>>>> > > > > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
>>>> > > > > > > > chiradeepv@gmail.com>
>>>> > > > > > > >     >> wrote:
>>>> > > > > > > >     >> >>>
>>>> > > > > > > >     >> >>> > Sounds workable. The downloader code in the
>>>> > > > > > > > SSVM
>>>> > > won't
>>>> > > > > > > follow
>>>> > > > > > > >     >> >>> > redirects I think.
>>>> > > > > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
>>>> > > > > > > 5511065fc20787619d
>>>> > > > > > > > 9cd0444
>>>> > > > > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/
>>>> > > storage/template/
>>>> > > > > > > >     >> >>> > HttpTemplateDownloader.java#L93
>>>> > > > > > > >     >> >>> > https://goo.gl/dSi0r5
>>>> > > > > > > >     >> >>> >  Might need to add
>>>> > > > > > > >     >> >>> > client.setRedirectStrategy(new
>>>> > > LaxRedirectStrategy());
>>>> > > > > > > >     >> >>> >
>>>> > > > > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
>>>> > > > > > > >     >> >>> > <ws...@cloudops.com>
>>>> > > > > > > >     >> >>> > wrote:
>>>> > > > > > > >     >> >>> >
>>>> > > > > > > >     >> >>> > > We haven't opened a ticket yet because we
>>>> > > > > > > > don't
>>>> > > > have a
>>>> > > > > > > > strategy
>>>> > > > > > > >     >> yet.
>>>> > > > > > > >     >> >>> > >
>>>> > > > > > > >     >> >>> > > What do you guys think of this:
>>>> > > > > > > >     >> >>> > > - We setup a new github repo in the
> 'apache'
>>>> > > > > > > > org
>>>> > > > which
>>>> > > > > > > > consists
>>>> > > > > > > >     >> >>> > > of a
>>>> > > > > > > >     >> >>> > single
>>>> > > > > > > >     >> >>> > > file with a list of active/supported
>> mirrors.
>>>> > > > > > > >     >> >>> > > - I write a small web server, distributed
> as
>>>> > > > > > > > a
>>>> > > > binary,
>>>> > > > > > > > which can
>>>> > > > > > > >     >> >>> > > be
>>>> > > > > > > >     >> >>> > hosted
>>>> > > > > > > >     >> >>> > > by ASF Infra.  This web server will query
>> the
>>>> > > > current
>>>> > > > > > list
>>>> > > > > > > > of
>>>> > > > > > > >     >> >>> > > mirrors
>>>> > > > > > > >     >> >>> and
>>>> > > > > > > >     >> >>> > > will select one and then do a 302 redirect
>> to
>>>> > > > > > > > that
>>>> > > > > > mirror.
>>>> > > > > > > >     >> >>> > >
>>>> > > > > > > >     >> >>> > > The act of 'choosing' a mirror could be
> done
>>>> > > > > > > > in a
>>>> > > > > number
>>>> > > > > > > of
>>>> > > > > > > > ways.
>>>> > > > > > > >     >> >>> > > - If we want to define an order, then it
>>>> > > > > > > > could
>>>> > > just
>>>> > > > > try
>>>> > > > > > > > from the
>>>> > > > > > > >     >> >>> > > top of
>>>> > > > > > > >     >> >>> > the
>>>> > > > > > > >     >> >>> > > list and work its way down.  It would curl
>>>> > > > > > > > the
>>>> > > > target
>>>> > > > > to
>>>> > > > > > > > make
>>>> > > > > > > >     >> >>> > > sure it
>>>> > > > > > > >     >> >>> > gets
>>>> > > > > > > >     >> >>> > > a 200 and if it does, it would do a 302
>>>> > redirect.
>>>> > > > > > > >     >> >>> > > - Or, if we want to distribute the load
>>>> > > > > > > > across the
>>>> > > > > > > mirrors,
>>>> > > > > > > > we
>>>> > > > > > > >     >> >>> > > could
>>>> > > > > > > >     >> >>> pick
>>>> > > > > > > >     >> >>> > > from the list randomly.  Again, doing a
> curl
>>>> > > > > > > > to
>>>> > > > verify
>>>> > > > > > the
>>>> > > > > > > > mirror
>>>> > > > > > > >     >> >>> > > is up
>>>> > > > > > > >     >> >>> > and
>>>> > > > > > > >     >> >>> > > then doing a redirect.
>>>> > > > > > > >     >> >>> > > - If we want to get fancy, we could do a
>>>> > > > > > > > reverse
>>>> > > IP
>>>> > > > > > lookup
>>>> > > > > > > > and
>>>> > > > > > > >     >> >>> > > try to
>>>> > > > > > > >     >> >>> > match
>>>> > > > > > > >     >> >>> > > the requester with their closest
>> geographical
>>>> > > > mirror.
>>>> > > > > > > >     >> >>> > >
>>>> > > > > > > >     >> >>> > > Thoughts?
>>>> > > > > > > >     >> >>> > >
>>>> > > > > > > >     >> >>> > > *Will STEVENS*
>>>> > > > > > > >     >> >>> > > Lead Developer
>>>> > > > > > > >     >> >>> > >
>>>> > > > > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
>>>> > > > > > > >     >> >>> > >
>>>> > > > > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep
>>>> > > Vittal <
>>>> > > > > > > >     >> >>> chiradeepv@gmail.com
>>>> > > > > > > >     >> >>> > >
>>>> > > > > > > >     >> >>> > > wrote:
>>>> > > > > > > >     >> >>> > >
>>>> > > > > > > >     >> >>> > > > What steps are needed to set up a mirror?
>>>> > > > > > > > What
>>>> > > > does
>>>> > > > > > > Infra
>>>> > > > > > > > need
>>>> > > > > > > >     >> to do?
>>>> > > > > > > >     >> >>> > Has
>>>> > > > > > > >     >> >>> > > > anybody filed a ticket with Infra?
>>>> > > > > > > >     >> >>> > > >
>>>> > > > > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja
>>>> > Pullela <
>>>> > > > > > > >     >> >>> > > > raja.pullela@accelerite.com>
>>>> > > > > > > >     >> >>> > > > wrote:
>>>> > > > > > > >     >> >>> > > >
>>>> > > > > > > >     >> >>> > > > > Hi will,
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > > > I believe, we didn’t get to close
>>>> > > > > > > > ‘getting a
>>>> > > > > mirror
>>>> > > > > > on
>>>> > > > > > > > Apache’
>>>> > > > > > > >     >> >>> > because
>>>> > > > > > > >     >> >>> > > we
>>>> > > > > > > >     >> >>> > > > > needed someone on the Apache Infra side
>>>> > > > > > > > to
>>>> > > close
>>>> > > > > > this.
>>>> > > > > > > > BTW,
>>>> > > > > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux
>>>> > > > > > > > manages
>>>> > > > this?)
>>>> > > > > > has
>>>> > > > > > > >     >> >>> > > > > all/most of
>>>> > > > > > > >     >> >>> > the
>>>> > > > > > > >     >> >>> > > > > content.  Once we can close on the
>> Apache
>>>> > > mirror
>>>> > > > > for
>>>> > > > > > > > hosting
>>>> > > > > > > >     >> >>> > > > > the
>>>> > > > > > > >     >> >>> > > > content, I
>>>> > > > > > > >     >> >>> > > > > can help assist getting the content
>> there.
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > > > For now, we have replicated the
>>>> > > > > download.cloud.com
>>>> > > > > > > > content to
>>>> > > > > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
>>>> > > > > > > >     >> >>> > > > > Also, we are working on a set of
>>>> > > steps/procedure
>>>> > > > > to
>>>> > > > > > > > help with
>>>> > > > > > > >     >> >>> > > > > this change.  I will update everyone in
>>>> > > > > > > > about
>>>> > > a
>>>> > > > > > week’s
>>>> > > > > > > > time
>>>> > > > > > > >     >> >>> > > > > on the
>>>> > > > > > > >     >> >>> > details.
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > > > Best,
>>>> > > > > > > >     >> >>> > > > > Raja Pullela
>>>> > > > > > > >     >> >>> > > > > Engineering Team,
>>>> > > > > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa
>>>> > > > > > > > Clara,
>>>> > > > CA,
>>>> > > > > > > 95054
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "
>>>> > > williamstevens@gmail.com
>>>> > > > > on
>>>> > > > > > > > behalf of
>>>> > > > > > > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com
>>>> > > > > > > > on
>>>> > > > behalf
>>>> > > > > > of
>>>> > > > > > > >     >> >>> > wstevens@cloudops.com>
>>>> > > > > > > >     >> >>> > > > > wrote:
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > > > unfortunately the template mirror
>>>> > > > > > > > conversation
>>>> > > > got
>>>> > > > > > > > caught up
>>>> > > > > > > >     >> >>> > > > > in
>>>> > > > > > > >     >> >>> > details
>>>> > > > > > > >     >> >>> > > > and
>>>> > > > > > > >     >> >>> > > > > nobody took the lead on implementing a
>>>> > > solution.
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > > > citrix has been pinging me every couple
>>>> > > > > > > > months
>>>> > > > to
>>>> > > > > > say
>>>> > > > > > > > 'dude,
>>>> > > > > > > >     >> >>> > > > > we
>>>> > > > > > > >     >> >>> need
>>>> > > > > > > >     >> >>> > to
>>>> > > > > > > >     >> >>> > > > > remove the dependency on
>>>> > > > > > > > download.citrix.com
>>>> > > ',
>>>> > > > > but
>>>> > > > > > i
>>>> > > > > > > > have not
>>>> > > > > > > >     >> >>> > > > > had
>>>> > > > > > > >     >> >>> > the
>>>> > > > > > > >     >> >>> > > > > cycles to get in and solve the problem.
>>>> > > > > > > > the
>>>> > > > > > shutdown
>>>> > > > > > > > of that
>>>> > > > > > > >     >> >>> > > > > is
>>>> > > > > > > >     >> >>> > > imminent
>>>> > > > > > > >     >> >>> > > > > right now, so we need to solve it asap.
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > > > *Will STEVENS*
>>>> > > > > > > >     >> >>> > > > > Lead Developer
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul
>>>> > Angus <
>>>> > > > > > > >     >> >>> > paul.angus@shapeblue.com
>>>> > > > > > > >     >> >>> > > >
>>>> > > > > > > >     >> >>> > > > > wrote:
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > > > > Hi Nathan,
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > > Ideally, if you put the template
>>>> > > > > > > > location in
>>>> > > > (or
>>>> > > > > > use
>>>> > > > > > > a
>>>> > > > > > > >     >> >>> > > > > > template
>>>> > > > > > > >     >> >>> > > defined
>>>> > > > > > > >     >> >>> > > > > > in)  test_data.py then the actual
>>>> > > > > > > > location
>>>> > > can
>>>> > > > > be
>>>> > > > > > > >     >> >>> > > > > > overridden by
>>>> > > > > > > >     >> >>> > > anyone
>>>> > > > > > > >     >> >>> > > > > > testing.
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > > For Trillian, we've copied all of the
>>>> > > > templates
>>>> > > > > > that
>>>> > > > > > > > people
>>>> > > > > > > >     >> >>> > > > > > have
>>>> > > > > > > >     >> >>> > > define
>>>> > > > > > > >     >> >>> > > > > to
>>>> > > > > > > >     >> >>> > > > > > a local repo and then replace the
> URLs
>>>> > > > > > > > in test_data.py to
>>>> > > > > > > >     >> >>> > > > > > reduce
>>>> > > > > > > >     >> >>> > > > > bandwidth
>>>> > > > > > > >     >> >>> > > > > > use and download times.
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > > Ie:
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > >             "bootableIso":
>>>> > > > > > > >     >> >>> > > > > >                 {
>>>> > > > > > > >     >> >>> > > > > >                     "displaytext":
>> "Test
>>>> > > > > Bootable
>>>> > > > > > > > ISO",
>>>> > > > > > > >     >> >>> > > > > >                     "name":
> "testISO",
>>>> > > > > > > >     >> >>> > > > > >                     "bootable": True,
>>>> > > > > > > >     >> >>> > > > > >                     "ispublic":
> False,
>>>> > > > > > > >     >> >>> > > > > >                     "url": "{{
>>>> > > > > > > marvin_images_location
>>>> > > > > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
>>>> > > > > > > >     >> >>> > > > > >                     "ostype": 'Other
>>>> Linux
>>>> > > > > > > (64-bit)',
>>>> > > > > > > >     >> >>> > > > > >                     "mode":
>>>> > 'HTTP_DOWNLOAD'
>>>> > > > > > > >     >> >>> > > > > >         },
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > > I thought that we had come up with a
>>>> > > solution
>>>> > > > > for
>>>> > > > > > > >     >> >>> > download.cloud.com
>>>> > > > > > > >     >> >>> > > ,
>>>> > > > > > > >     >> >>> > > > by
>>>> > > > > > > >     >> >>> > > > > > having a mirrorlist hosted in
>> Community
>>>> > > Apache
>>>> > > > > > > > 'space' with
>>>> > > > > > > >     >> >>> anyone
>>>> > > > > > > >     >> >>> > > able
>>>> > > > > > > >     >> >>> > > > > to
>>>> > > > > > > >     >> >>> > > > > > out themselves forward as a mirror.
>>>> > > > > > > >     >> >>> > > > > > But I must admit I lost track of
>>>> > > > > > > > whether
>>>> > > > anyone
>>>> > > > > > made
>>>> > > > > > > > the
>>>> > > > > > > >     >> >>> requisite
>>>> > > > > > > >     >> >>> > > > > changes
>>>> > > > > > > >     >> >>> > > > > > in code....
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > > Kind regards,
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > > Paul Angus
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > > paul.angus@shapeblue.com
>>>> > > > > > > >     >> >>> > > > > > www.shapeblue.com
>>>> > > > > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden,
>> London
>>>> > > WC2N
>>>> > > > > > 4HSUK
>>>> > > > > > > >     >> >>> > > > > > @shapeblue
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > > -----Original Message-----
>>>> > > > > > > >     >> >>> > > > > > From: williamstevens@gmail.com
>>>> > > > > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
>>>> > > > > > > >     >> >>> > On
>>>> > > > > > > >     >> >>> > > > > > Behalf Of Will Stevens
>>>> > > > > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
>>>> > > > > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
>>>> > > > > > > >     >> >>> > > > > > Subject: Re: Modern template hosting
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > > this is a hard questions.  in
> general,
>>>> > > > > > > > we
>>>> > > > should
>>>> > > > > > be
>>>> > > > > > > > setting
>>>> > > > > > > >     >> >>> > > > > > up a
>>>> > > > > > > >     >> >>> > > mirror
>>>> > > > > > > >     >> >>> > > > > on
>>>> > > > > > > >     >> >>> > > > > > some cloudstack/apache domain and
> then
>>>> > > mirror
>>>> > > > to
>>>> > > > > > > other
>>>> > > > > > > >     >> >>> > > > > > provided
>>>> > > > > > > >     >> >>> > > > > templates.
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > > we MUST come up with a solution to
>>>> > deprecate
>>>> > > > > > > >     >> >>> > > > > > 'download.cloud.com
>>>> > > > > > > >     >> >>> ',
>>>> > > > > > > >     >> >>> > > > that
>>>> > > > > > > >     >> >>> > > > > > is going to be going away any day
> now.
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > > i don't know the right way to solve
>>>> > > > > > > > this to
>>>> > > be
>>>> > > > > > > > honest, but
>>>> > > > > > > >     >> >>> > > > > > if you
>>>> > > > > > > >     >> >>> > > have
>>>> > > > > > > >     >> >>> > > > > > ideas, i am willing to help.
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > > *Will STEVENS*
>>>> > > > > > > >     >> >>> > > > > > Lead Developer
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM,
>>>> > > > > > > > Nathan
>>>> > > > > Johnson <
>>>> > > > > > > >     >> >>> njohnson@ena.com
>>>> > > > > > > >     >> >>> > >
>>>> > > > > > > >     >> >>> > > > > wrote:
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > > > > So not to re-open a can of worms,
>> but
>>>> > > > > > > > I’m
>>>> > > > in a
>>>> > > > > > > > situation
>>>> > > > > > > >     >> >>> > > > > > > where
>>>> > > > > > > >     >> >>> I
>>>> > > > > > > >     >> >>> > > need
>>>> > > > > > > >     >> >>> > > > > > > to come up with a Marvin component
>>>> > > > > > > > test
>>>> > > that
>>>> > > > > > > > depends on a
>>>> > > > > > > >     >> >>> > template
>>>> > > > > > > >     >> >>> > > > > > > based on a kernel that’s relatively
>>>> > > > > > > > new,
>>>> > > > i.e.,
>>>> > > > > > > > newer than
>>>> > > > > > > >     >> >>> Centos
>>>> > > > > > > >     >> >>> > > 5.3
>>>> > > > > > > >     >> >>> > > > /
>>>> > > > > > > >     >> >>> > > > > > Ubuntu 10.04 .
>>>> > > > > > > >     >> >>> > > > > > > I see openvm.eu has a suitable
>>>> > > > > > > > template
>>>> > > > > (Ubuntu
>>>> > > > > > > > 16.0.4
>>>> > > > > > > >     >> >>> > > > > > > for
>>>> > > > > > > >     >> >>> KVM),
>>>> > > > > > > >     >> >>> > > but
>>>> > > > > > > >     >> >>> > > > > > > from looking at the thread
>> "Migrating
>>>> > > > > CloudStack
>>>> > > > > > > > content
>>>> > > > > > > >     >> >>> > > > > > > from download.cloud.com” it looks
>>>> > > > > > > > like
>>>> > > > there
>>>> > > > > is
>>>> > > > > > > >     >> >>> > > > > > > resistance to using
>>>> > > > > > > >     >> >>> > > this
>>>> > > > > > > >     >> >>> > > > at
>>>> > > > > > > >     >> >>> > > > > > > least for hosting system vm
>> templates
>>>> > > > > > > > over
>>>> > > > > > > concerns
>>>> > > > > > > > of
>>>> > > > > > > >     >> >>> > neutrality.
>>>> > > > > > > >     >> >>> > > > > > > Would this be suitable for a
>>>> > > > > > > > component
>>>> > > test?
>>>> > > > > If
>>>> > > > > > > > not,
>>>> > > > > > > >     >> >>> > > > > > > what is a
>>>> > > > > > > >     >> >>> > > > > “blessed”
>>>> > > > > > > >     >> >>> > > > > > > template location?
>>>> > > > > > > >     >> >>> > > > > > >
>>>> > > > > > > >     >> >>> > > > > > > Thanks in advance!
>>>> > > > > > > >     >> >>> > > > > > >
>>>> > > > > > > >     >> >>> > > > > > > Nathan Johnson
>>>> > > > > > > >     >> >>> > > > > > > R&D Engineer
>>>> > > > > > > >     >> >>> > > > > > >
>>>> > > > > > > >     >> >>> > > > > > >
>>>> > > > > > > >     >> >>> > > > > > >
>>>> > > > > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12
>>>> > > > Nashville,
>>>> > > > > TN
>>>> > > > > > > > 37211
>>>> > > > > > > >     >> >>> > > > > > > General Office: 615-312-6000
>>>> > > > > > > >     >> >>> > > > > > >
>>>> > > > > > > >     >> >>> > > > > > > website | blog | support
>>>> > > > > > > >     >> >>> > > > > > >
>>>> > > > > > > >     >> >>> > > > > > >
>>>> > > > > > > >     >> >>> > > > > > >
>>>> > > > > > > >     >> >>> > > > > > >
>>>> > > > > > > >     >> >>> > > > > > >
>>>> > > > > > > >     >> >>> > > > > > >
>>>> > > > > > > >     >> >>> > > > > > >
>>>> > > > > > > >     >> >>> > > > > >
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > > > DISCLAIMER
>>>> > > > > > > >     >> >>> > > > > ==========
>>>> > > > > > > >     >> >>> > > > > This e-mail may contain privileged and
>>>> > > > > confidential
>>>> > > > > > > >     >> >>> > > > > information
>>>> > > > > > > >     >> >>> which
>>>> > > > > > > >     >> >>> > > is
>>>> > > > > > > >     >> >>> > > > > the property of Accelerite, a
> Persistent
>>>> > > Systems
>>>> > > > > > > > business. It
>>>> > > > > > > >     >> >>> > > > > is
>>>> > > > > > > >     >> >>> > > intended
>>>> > > > > > > >     >> >>> > > > > only for the use of the individual or
>>>> > > > > > > > entity
>>>> > > to
>>>> > > > > > which
>>>> > > > > > > > it is
>>>> > > > > > > >     >> >>> > addressed.
>>>> > > > > > > >     >> >>> > > If
>>>> > > > > > > >     >> >>> > > > > you are not the intended recipient, you
>>>> > > > > > > > are
>>>> > > not
>>>> > > > > > > > authorized to
>>>> > > > > > > >     >> >>> > > > > read,
>>>> > > > > > > >     >> >>> > > > retain,
>>>> > > > > > > >     >> >>> > > > > copy, print, distribute or use this
>>>> message.
>>>> > > If
>>>> > > > > you
>>>> > > > > > > have
>>>> > > > > > > >     >> >>> > > > > received
>>>> > > > > > > >     >> >>> > this
>>>> > > > > > > >     >> >>> > > > > communication in error, please notify
>> the
>>>> > > sender
>>>> > > > > and
>>>> > > > > > > > delete
>>>> > > > > > > >     >> >>> > > > > all
>>>> > > > > > > >     >> >>> > copies
>>>> > > > > > > >     >> >>> > > of
>>>> > > > > > > >     >> >>> > > > > this message. Accelerite, a Persistent
>>>> > > > > > > > Systems
>>>> > > > > > > business
>>>> > > > > > > > does
>>>> > > > > > > >     >> >>> > > > > not
>>>> > > > > > > >     >> >>> > accept
>>>> > > > > > > >     >> >>> > > > any
>>>> > > > > > > >     >> >>> > > > > liability for virus infected mails.
>>>> > > > > > > >     >> >>> > > > >
>>>> > > > > > > >     >> >>> > > >
>>>> > > > > > > >     >> >>> > >
>>>> > > > > > > >     >> >>> >
>>>> > > > > > > >     >> >>>
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >abhinandan.prateek@shapeblue.com
>>>> > > > > > > >     >> >www.shapeblue.com
>>>> > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N
> 4HSUK
>>>> > > > > @shapeblue
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >> >
>>>> > > > > > > >     >>
>>>> > > > > > > >     >> abhinandan.prateek@shapeblue.com
>>>> > > > > > > >     >> www.shapeblue.com
>>>> > > > > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>>> > > > > > > >     >> @shapeblue
>>>> > > > > > > >     >>
>>>> > > > > > > >     >>
>>>> > > > > > > >     >>
>>>> > > > > > > >     >>
>>>> > > > > > > >     >
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > > > daan.hoogland@shapeblue.com
>>>> > > > > > > > www.shapeblue.com
>>>> > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>>> > > > > > > > @shapeblue
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > >
>>>> > > > > > >
>>>> > > > > > > --
>>>> > > > > > > Rafael Weingärtner
>>>> > > > > > >
>>>> > > > > >
>>>> > > > > >
>>>> > > > > >
>>>> > > > > > --
>>>> > > > > > Rafael Weingärtner
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>>

Re: Modern template hosting

Posted by Will Stevens <wi...@gmail.com>.

Erik, yes I think this is the right approach and covers the main problem
case. I can start the conversation with infra unless you would like to. Let
me know.

We did get some statistics from the other mirrors last year, but I don't
think we ever had visibility into the cloud.com repository traffic.

On Mar 3, 2017 6:42 PM, "Erik Weber" <te...@gmail.com> wrote:

It'll most likely only be an issue in the case where a mirror goes
down without any immediate chance of getting back up, right?
Could we check with ASF Infra beforehand if such case is OK to warrant
an urgent Jira ticket to resolve - should it happen?

We can document all the mirrors and provide the howtos for manually
downloading and seeding the image as a backup.

Do we have any traffic statistics for the current solution, or is that
not available at S3?

--
Erik

On Sat, Mar 4, 2017 at 12:28 AM, Will Stevens <wi...@gmail.com>
wrote:
> Yes Erik, I agree with you. The only thing that could add complexity in
> this is the fact that we don't control the domain, the ASF does (from what
> I understand).  Do we expect this to be managed by ASF infra or would
there
> be another way? Ideally the domain is community controlled so we can adapt
> in case of change without long delays.
>
> I am 100% in agreement that this is the shortest path and likely to be
good
> enough.
>
> On Mar 3, 2017 6:19 PM, "Erik Weber" <te...@gmail.com> wrote:
>
> IMHO; implement it as simple as possible, use DNS RR and assume/expect
> mirror hosts to use the same path.
>
>
> Yes I know; DNS RR isn't bulletproof, but as already mentioned in this
> thread - mirrors rarely go down and they aren't used all that often
> (primarily seeding).
> We can keep a low TTL on the record so that we're able to remove a
> mirror that goes down for a significant period of time (more than
> hours).
>
> And yes I know, some operators might find it a burden having to set up
> a new vhost with another document root, but tbh if they don't care
> enough to do it we'll manage without that mirror.
>
> --
> Erik
>
> On Sat, Mar 4, 2017 at 12:09 AM, Will Stevens <wi...@gmail.com>
> wrote:
>> I agree with Paul. Look at the list of things they have to
>> learn/master/care about. We don't want to add to that list, we want to
>> remove from that list. Make the project/product more accessible. Reduce
> the
>> barrier to entry...
>>
>> On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:
>>
>>> Seriously?
>>> apt-get install apache2
>>>
>>> To install CloudStack, they need to know
>>> - DB installation, perhaps some SQL
>>> - VLAN configuration on their switches
>>> - Ins-and-outs between port forwarding, static ips,
>>> - NFS
>>> - package management
>>> - VHDs, qcow2. vmdk
>>> - hyperviosrs
>>> - and on and on.
>>>
>>> If they can figure all that out and not figure out a web server, they
can
>>> always come to the mailing list.
>>>
>>>
>>>
>>>
>>> On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <pa...@shapeblue.com>
>>> wrote:
>>>
>>> > The issue is not with supporting highly experienced cloud operators.
It
>>> is
>>> > an issue for new users and other 'relatively' inexperienced operators.
>>> >
>>> > I have helped enough newbies and cloud operators who have been running
>>> > their cloud for a while to know that the barriers to entry are too
high
>>> as
>>> > they are. And telling anyone that they need to create a web server so
>>> that
>>> > they can add their initial template to get started or in order to
> create
>>> a
>>> > new zone, just isn't going to fly.
>>> >
>>> > I'm a huge advocate of the 'download.cloudstack.org' endpoint which
the
>>> > community can add/remove mirrors to or from for system VMs or built-in
>>> > templates.  Can the same system be used for binary repos ? although
> there
>>> > is an added complication of redist vs no-redist there...
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > paul.angus@shapeblue.com
>>> > www.shapeblue.com
>>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>> > @shapeblue
>>> >
>>> >
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: Chiradeep Vittal [mailto:chiradeepv@gmail.com]
>>> > Sent: 03 March 2017 18:28
>>> > To: dev <de...@cloudstack.apache.org>
>>> > Subject: Re: Modern template hosting
>>> >
>>> > I do feel like this is early optimization. Mirrors rarely fail. I'd
>>> expect
>>> > a single web server hosted on Apache Infra without any monitors to
fail
>>> > more often than a mirror. We already expect Wido's systemvm repository
> to
>>> > be up all the time. And it has been. Similarly, I don't believe Nux's
>>> > repository has ever been down. And if Accelerite wants to host on S3,
>>> that
>>> > one is pretty solid as well.
>>> >
>>> > This is an infrequent operation in a cloud. After the cloud is
> installed,
>>> > the download servers are only needed for a new zone. If we trust the
> user
>>> > to run a cloud, surely he/she can run a web server to serve some
> built-in
>>> > templates. And if her cloud is successful, she needs to figure out how
> to
>>> > host her templates anyway and not rely on 3rd parties.
>>> >
>>> >
>>> > On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <ws...@cloudops.com>
>>> > wrote:
>>> >
>>> > > 1) If the legacy implementations do not support redirects, that does
>>> > > cause a problem.  A potential solution in that case is to have the
> web
>>> > > server actually proxy the download, but that is not ideal and I
would
>>> > > like to avoid it if possible.  Thanks for bringing that up
Chiradeep.
>>> > >
>>> > > 2) I think we need to have a single URL which people can target.
> Once
>>> > > they make the switch to the new URL, we want the implementation to
be
>>> > > able to handle mirror failures without affecting the end client.  We
>>> > > want to avoid the situation where an ACS user will ever have to
> change
>>> > > this URL more than once.  Mirror failures SHOULD NOT affect the ACS
>>> > > users assuming there is still at least one mirror who can serve the
>>> > requested resource.
>>> > >
>>> > > These are obviously my personal opinions and others will probably
> have
>>> > > differing opinions.
>>> > >
>>> > > *Will STEVENS*
>>> > > Lead Developer
>>> > >
>>> > > <https://goo.gl/NYZ8KK>
>>> > >
>>> > > On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal
>>> > > <ch...@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > 1. If you are targeting legacy installations, they are not able to
>>> > > > follow redirects. The line of code that added this capability was
>>> > > > added on
>>> > > 11/16.
>>> > > > 2. If you trust the users to edit the database to change the URL,
>>> > > > you can trust them to change it to anything. Just document a known
>>> > > > list of good template locations.
>>> > > >
>>> > > > On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <
wstevens@cloudops.com
>>
>>> > > > wrote:
>>> > > >
>>> > > > > So the main issue I see with this is the following.
>>> > > > >
>>> > > > > This implementation is designed to target legacy installations
>>> > > > > which
>>> > > will
>>> > > > > be affected when download.cloud.com disappears.  These people
> will
>>> > > need
>>> > > > to
>>> > > > > make a DB change to replace the 'download.cloud.com' with some
>>> > > > > other
>>> > > url
>>> > > > > (maybe 'download.cloudstack.org').  Once the DB has been
updated,
>>> > > > > we
>>> > > can
>>> > > > > not expect anything else of the client, they need to be able to
>>> > > > > just continue operation as they were without any need to rebuild
>>> > > > > or upgrade their ACS.
>>> > > > >
>>> > > > > If we try to force the decision to the client, then we break the
>>> > > > > legacy implementations.  Since the SSVM is likely going to be
the
>>> > > > > client in
>>> > > some
>>> > > > > cases, and since it does not already have the logic to handle
the
>>> > > > > 300 approach correctly, I feel like it is not a viable solution
>>> > > > > for the
>>> > > > legacy
>>> > > > > deployments.
>>> > > > >
>>> > > > > In addition to that.  I don't think the client is in any better
>>> > > position
>>> > > > to
>>> > > > > make the mirror decision than I am.  I am likely able to make a
>>> > > > > more 'educated' decision than the client would be able to
because
>>> > > > > I can do different tests on the endpoint(s) before making a
final
>>> > decision.
>>> > > > >
>>> > > > > Does this make sense or am I missing something obvious?
>>> > > > >
>>> > > > > *Will STEVENS*
>>> > > > > Lead Developer
>>> > > > >
>>> > > > > <https://goo.gl/NYZ8KK>
>>> > > > >
>>> > > > > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
>>> > > > > rafaelweingartner@gmail.com> wrote:
>>> > > > >
>>> > > > > > I think I understood you, but I did not understand you.
>>> > > > > >
>>> > > > > > Let me see if I can create an example to illustrate how we
> could
>>> > > > > > do
>>> > > > using
>>> > > > > > the HTTP 300 code.
>>> > > > > >
>>> > > > > > The HTTP 300 code indicates clients that he/she/it has
multiple
>>> > > choices
>>> > > > > to
>>> > > > > > access the requested resource. If the request is a HEAD, this
> is
>>> > > > > > the
>>> > > > only
>>> > > > > > thing that the server returns. If the request if a GET, then
> the
>>> > > server
>>> > > > > > returns the choices for the client. The response type is
> defined
>>> > > > > > on
>>> > > the
>>> > > > > > server based on the Content-Type, user agent and others.
>>> > > > > >
>>> > > > > > Let’s say the server receives a GET request and informs the
>>> > > > content-type
>>> > > > > as
>>> > > > > > JSON (application/JSON). Your application could get the
mirrors
>>> > > > > > list,
>>> > > > do
>>> > > > > > the appending process for the requested resource path, and
then
>>> > > return
>>> > > > a
>>> > > > > > JSON array with the possible mirror the client can use. Then,
>>> > > > > > the
>>> > > > client
>>> > > > > > gets this list and does the processing required to select a
>>> mirror.
>>> > > > > >
>>> > > > > > Now let’s say that a human access the link using a browser.
>>> > > > > > Then, the server should receive something like (text/plain) as
>>> > > > > > the
>>> > > content-type;
>>> > > > I
>>> > > > > > think depending on the browser this field may differ a little
>>> > > > > > (it
>>> > > would
>>> > > > > > require some checking). Anyways, the server detects that it is
> a
>>> > > > “human”
>>> > > > > > requesting the resource, then we could serve an HTML page with
> a
>>> > > > > Javascript
>>> > > > > > that uses the mirror list. This Javascript could do some
>>> > > > > > checking and choose the best mirror for that specific.
>>> > > > > >
>>> > > > > > Why do I think this approach is interesting?
>>> > > > > > The application you developed would be used only to retrieve
>>> > > > > > valid
>>> > > and
>>> > > > > > trusted sources of system VM images (a trusted repository for
>>> > > > > > mirrors
>>> > > > > that
>>> > > > > > we as a community vouch for).  Also, we shift the decision
>>> > > > > > process regarding mirrors from the server to the client. Then,
>>> > > > > > it is up to
>>> > > > > clients
>>> > > > > > to select mirrors, and not up to us (or some of our
>>> > implementations).
>>> > > > > >
>>> > > > > > Did this help?
>>> > > > > > BTW: I am not saying I am against the way you proposed, which
>>> > > > > > would
>>> > > > work
>>> > > > > > fine. It is merely a suggestion using a different perspective.
>>> > > > > >
>>> > > > > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
>>> > > > williamstevens@gmail.com>
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > > @rafael: in general, I don't think the client should ever be
>>> > > making a
>>> > > > > > > choice about a mirror. We have to assume we are working with
> a
>>> > > > scripted
>>> > > > > > > application and anything hard coded in that implementation
on
>>> > > > > > > the
>>> > > > > client
>>> > > > > > > side is a risk.
>>> > > > > > >
>>> > > > > > > I may not be understanding the use of the 300 approach, so I
>>> > > > > > > will
>>> > > > > > research
>>> > > > > > > it to see if I can make it fit.
>>> > > > > > >
>>> > > > > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
>>> > > > > > rafaelweingartner@gmail.com>
>>> > > > > > > wrote:
>>> > > > > > >
>>> > > > > > > Will great job.
>>> > > > > > >
>>> > > > > > > I had the same doubt as Daan.
>>> > > > > > >
>>> > > > > > > Have you considered using HTTP 300 working mode? Then, we
>>> > > > > > > could let
>>> > > > the
>>> > > > > > > client decide which mirror is the best(closest?). Section
>>> > “10.3.1”
>>> > > of
>>> > > > > [1]
>>> > > > > > > talks about it; this would put the pressure on deciding upon
> a
>>> > > mirror
>>> > > > > on
>>> > > > > > > the client side, but I think it would be fair.
>>> > > > > > >
>>> > > > > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
>>> > > > > > >
>>> > > > > > > PS: You really like very short variables names!
>>> > > > > > >
>>> > > > > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
>>> > > > > > daan.hoogland@shapeblue.com
>>> > > > > > > >
>>> > > > > > > wrote:
>>> > > > > > >
>>> > > > > > > > Nice little thing Will,
>>> > > > > > > >
>>> > > > > > > > One question: if I read the code correctly it ‘go’es and
>>> > > > > > > > tries
>>> > > all
>>> > > > > > > mirrors
>>> > > > > > > > at once and whichever responses first is redirected to the
>>> > > client.
>>> > > > > This
>>> > > > > > > > might well be the same every time. This might be the one
>>> > > > > > > > closest
>>> > > to
>>> > > > > the
>>> > > > > > > > mirror302 and not to the client. Is that correct or did I
>>> > > > > > > > miss a
>>> > > > > > weighing
>>> > > > > > > > algorithm hidden in there?
>>> > > > > > > >
>>> > > > > > > > Good coding,
>>> > > > > > > >
>>> > > > > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org>
wrote:
>>> > > > > > > >
>>> > > > > > > >     Hey All,
>>> > > > > > > >     Please review this repo and tell me what you think:
>>> > > > > > > >     https://github.com/swill/mirror302
>>> > > > > > > >
>>> > > > > > > >     Let me know if you have questions or you would like me
>>> > > > > > > > to
>>> > > make
>>> > > > > > > changes
>>> > > > > > > >     before I bring this topic up with ASF Infra and open a
>>> > > > > > > > ticket
>>> > > > to
>>> > > > > > get
>>> > > > > > > > this
>>> > > > > > > >     implemented.
>>> > > > > > > >
>>> > > > > > > >     Cheers,
>>> > > > > > > >
>>> > > > > > > >     *Will Stevens*
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
>>> > > > > > wstevens@cloudops.com
>>> > > > > > > >
>>> > > > > > > > wrote:
>>> > > > > > > >
>>> > > > > > > >     > I am building a short term solution right now.  I
>>> > > > > > > > hopefully
>>> > > > > will
>>> > > > > > > > have the
>>> > > > > > > >     > building blocks in place by the weekend so we can
>>> > > > > > > > start
>>> > > > working
>>> > > > > > > with
>>> > > > > > > > Infra
>>> > > > > > > >     > to get it in place.
>>> > > > > > > >     >
>>> > > > > > > >     > We will have a web server which we can point
> something
>>> > like
>>> > > > > > > >     > downloads.cloudstack.org (or whatever url) at.  Then
>>> > > > > > > > we
>>> > > will
>>> > > > > > have
>>> > > > > > > > an ASF
>>> > > > > > > >     > repo which tracks a mirror list and exposes a static
>>> > > > > > > > site
>>> > > to
>>> > > > be
>>> > > > > > > able
>>> > > > > > > > to
>>> > > > > > > >     > query the mirror list.
>>> > > > > > > >     >
>>> > > > > > > >     > The web server will receive a request and will do a
>>> > > > > > > > 302
>>> > > > > redirect
>>> > > > > > to
>>> > > > > > > > the
>>> > > > > > > >     > appropriate resource on one of the mirrors.
>>> > > > > > > >     >
>>> > > > > > > >     > This gets us started.  Legacy environments will have
>>> > > > > > > > to do
>>> > > a
>>> > > > DB
>>> > > > > > > > change to
>>> > > > > > > >     > change from 'download.cloud.com' to '
>>> > > > downloads.cloudstack.org'
>>> > > > > > (or
>>> > > > > > > >     > whatever), but otherwise it should be pretty simple.
>>> > > > > > > >     >
>>> > > > > > > >     > We can improve how we deliver templates going
> forward,
>>> > > > > > > > but
>>> > > > this
>>> > > > > > > > obviously
>>> > > > > > > >     > requires some discussion still.  I will try to buy
us
>>> > > > > > > > some
>>> > > > time
>>> > > > > > > with
>>> > > > > > > > an
>>> > > > > > > >     > implementation which solves our problems today...
>>> > > > > > > >     >
>>> > > > > > > >     > *Will STEVENS*
>>> > > > > > > >     > Lead Developer
>>> > > > > > > >     >
>>> > > > > > > >     > <https://goo.gl/NYZ8KK>
>>> > > > > > > >     >
>>> > > > > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
>>> > > > > > > >     > abhinandan.prateek@shapeblue.com> wrote:
>>> > > > > > > >     >
>>> > > > > > > >     >>
>>> > > > > > > >     >> Initial seeding is a manual step and that is not
>>> > > > > > > > going to
>>> > > > > change
>>> > > > > > > in
>>> > > > > > > > near
>>> > > > > > > >     >> future. A handy list of official places from where
>>> > > > > > > > these
>>> > > > > > templates
>>> > > > > > > > can be
>>> > > > > > > >     >> downloaded won’t harm this part of the
installation.
>>> > > > > > > > Just
>>> > > > > allow
>>> > > > > > > > anyone
>>> > > > > > > >     >> credible who follows democratic process to publish
>>> > > > > > > > their
>>> > > > > > templates
>>> > > > > > > > with
>>> > > > > > > >     >> apache’s blessings.
>>> > > > > > > >     >>
>>> > > > > > > >     >>
>>> > > > > > > >     >> Coming to urls hard coded in db, the popular
>>> > > > > > > > suggestion is
>>> > > > to
>>> > > > > > > point
>>> > > > > > > > them
>>> > > > > > > >     >> to mirrors. This looks good but this is going to
> take
>>> > > > > > > > some
>>> > > > > > effort
>>> > > > > > > > that has
>>> > > > > > > >     >> not materialised yet. Once this change is made it
>>> > > > > > > > will be
>>> > > > > going
>>> > > > > > in
>>> > > > > > > > some
>>> > > > > > > >     >> future release like 4.11; the problem still remains
>>> > > > > > > > for
>>> > > the
>>> > > > > > > > releases that
>>> > > > > > > >     >> are already out there. In case access to
>>> > > download.cloud.com
>>> > > > > is
>>> > > > > > > > dropped
>>> > > > > > > >     >> at some point we will have to document the
procedure
>>> > > > > > > > such
>>> > > > that
>>> > > > > > > > people know
>>> > > > > > > >     >> how to make these stable releases work. That is
> where
>>> > > > > > > > the
>>> > > > > > > procedure
>>> > > > > > > >     >> documented here https://shankerbalan.net/blog/
>>> > > > > > > >     >> seed-cloudstack-templates-offline/ should be
>>> > > > > > > > officially
>>> > > > > adapted
>>> > > > > > > > with a
>>> > > > > > > >     >> marketplace to pick up templates of choosing. Maybe
>>> > > > > > > > we can
>>> > > > > have
>>> > > > > > a
>>> > > > > > > > pre-setup
>>> > > > > > > >     >> script to make choices instead of written wiki.
Note
>>> > > > > > > > that
>>> > > > this
>>> > > > > > > will
>>> > > > > > > > be
>>> > > > > > > >     >> anyway needed by the older releases.
>>> > > > > > > >     >>
>>> > > > > > > >     >>
>>> > > > > > > >     >>
>>> > > > > > > >     >>
>>> > > > > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
>>> > > > paul.angus@shapeblue.com>
>>> > > > > > > > wrote:
>>> > > > > > > >     >>
>>> > > > > > > >     >> >-1
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >We are trying to get MORE people to use
CloudStack.
>>> > > > > > > > And
>>> > > > > > raising
>>> > > > > > > > the
>>> > > > > > > >     >> barrier to entry does nothing to help that.
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >Kind regards,
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >Paul Angus
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >paul.angus@shapeblue.com
>>> > > > > > > >     >> >www.shapeblue.com
>>> > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N
4HSUK
>>> > > > > > > >     >> >@shapeblue
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >-----Original Message-----
>>> > > > > > > >     >> >From: Abhinandan Prateek [mailto:
> abhinandan.prateek@
>>> > > > > > > shapeblue.com]
>>> > > > > > > >     >> >Sent: 01 March 2017 09:39
>>> > > > > > > >     >> >To: dev@cloudstack.apache.org
>>> > > > > > > >     >> >Subject: Re: Modern template hosting
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >+1
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >After seeing this discussion running in circles
>>> > > > > > > > several
>>> > > > > times,
>>> > > > > > I
>>> > > > > > > > think
>>> > > > > > > >     >> we should at least get started with the simplest
>>> option.
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
>>> > > > > > chiradeepv@gmail.com>
>>> > > > > > > > wrote:
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >>The alternative foolproof way is to simply not
>>> > > > > > > > provide
>>> > > the
>>> > > > > > > > automatic
>>> > > > > > > >     >> >>download.
>>> > > > > > > >     >> >>
>>> > > > > > > >     >> >>Just document the mirror list URL where the
>>> > > > > > > > templates
>>> > > can
>>> > > > be
>>> > > > > > > > found. I
>>> > > > > > > >     >> >>don't know why people reject this option.
>>> > > > > > > >     >> >>
>>> > > > > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
>>> > > > > > > templates-offline/
>>> > > > > > > >     >> >>
>>> > > > > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
>>> > > > > > > >     >> >><wi...@gmail.com>
>>> > > > > > > >     >> >>wrote:
>>> > > > > > > >     >> >>
>>> > > > > > > >     >> >>> OK. Thanks for the heads up.
>>> > > > > > > >     >> >>>
>>> > > > > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
>>> > > > > > > > chiradeepv@gmail.com>
>>> > > > > > > >     >> wrote:
>>> > > > > > > >     >> >>>
>>> > > > > > > >     >> >>> > Sounds workable. The downloader code in the
>>> > > > > > > > SSVM
>>> > > won't
>>> > > > > > > follow
>>> > > > > > > >     >> >>> > redirects I think.
>>> > > > > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
>>> > > > > > > 5511065fc20787619d
>>> > > > > > > > 9cd0444
>>> > > > > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/
>>> > > storage/template/
>>> > > > > > > >     >> >>> > HttpTemplateDownloader.java#L93
>>> > > > > > > >     >> >>> > https://goo.gl/dSi0r5
>>> > > > > > > >     >> >>> >  Might need to add
>>> > > > > > > >     >> >>> > client.setRedirectStrategy(new
>>> > > LaxRedirectStrategy());
>>> > > > > > > >     >> >>> >
>>> > > > > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
>>> > > > > > > >     >> >>> > <ws...@cloudops.com>
>>> > > > > > > >     >> >>> > wrote:
>>> > > > > > > >     >> >>> >
>>> > > > > > > >     >> >>> > > We haven't opened a ticket yet because we
>>> > > > > > > > don't
>>> > > > have a
>>> > > > > > > > strategy
>>> > > > > > > >     >> yet.
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> > > What do you guys think of this:
>>> > > > > > > >     >> >>> > > - We setup a new github repo in the
'apache'
>>> > > > > > > > org
>>> > > > which
>>> > > > > > > > consists
>>> > > > > > > >     >> >>> > > of a
>>> > > > > > > >     >> >>> > single
>>> > > > > > > >     >> >>> > > file with a list of active/supported
> mirrors.
>>> > > > > > > >     >> >>> > > - I write a small web server, distributed
as
>>> > > > > > > > a
>>> > > > binary,
>>> > > > > > > > which can
>>> > > > > > > >     >> >>> > > be
>>> > > > > > > >     >> >>> > hosted
>>> > > > > > > >     >> >>> > > by ASF Infra.  This web server will query
> the
>>> > > > current
>>> > > > > > list
>>> > > > > > > > of
>>> > > > > > > >     >> >>> > > mirrors
>>> > > > > > > >     >> >>> and
>>> > > > > > > >     >> >>> > > will select one and then do a 302 redirect
> to
>>> > > > > > > > that
>>> > > > > > mirror.
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> > > The act of 'choosing' a mirror could be
done
>>> > > > > > > > in a
>>> > > > > number
>>> > > > > > > of
>>> > > > > > > > ways.
>>> > > > > > > >     >> >>> > > - If we want to define an order, then it
>>> > > > > > > > could
>>> > > just
>>> > > > > try
>>> > > > > > > > from the
>>> > > > > > > >     >> >>> > > top of
>>> > > > > > > >     >> >>> > the
>>> > > > > > > >     >> >>> > > list and work its way down.  It would curl
>>> > > > > > > > the
>>> > > > target
>>> > > > > to
>>> > > > > > > > make
>>> > > > > > > >     >> >>> > > sure it
>>> > > > > > > >     >> >>> > gets
>>> > > > > > > >     >> >>> > > a 200 and if it does, it would do a 302
>>> > redirect.
>>> > > > > > > >     >> >>> > > - Or, if we want to distribute the load
>>> > > > > > > > across the
>>> > > > > > > mirrors,
>>> > > > > > > > we
>>> > > > > > > >     >> >>> > > could
>>> > > > > > > >     >> >>> pick
>>> > > > > > > >     >> >>> > > from the list randomly.  Again, doing a
curl
>>> > > > > > > > to
>>> > > > verify
>>> > > > > > the
>>> > > > > > > > mirror
>>> > > > > > > >     >> >>> > > is up
>>> > > > > > > >     >> >>> > and
>>> > > > > > > >     >> >>> > > then doing a redirect.
>>> > > > > > > >     >> >>> > > - If we want to get fancy, we could do a
>>> > > > > > > > reverse
>>> > > IP
>>> > > > > > lookup
>>> > > > > > > > and
>>> > > > > > > >     >> >>> > > try to
>>> > > > > > > >     >> >>> > match
>>> > > > > > > >     >> >>> > > the requester with their closest
> geographical
>>> > > > mirror.
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> > > Thoughts?
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> > > *Will STEVENS*
>>> > > > > > > >     >> >>> > > Lead Developer
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep
>>> > > Vittal <
>>> > > > > > > >     >> >>> chiradeepv@gmail.com
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> > > wrote:
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> > > > What steps are needed to set up a mirror?
>>> > > > > > > > What
>>> > > > does
>>> > > > > > > Infra
>>> > > > > > > > need
>>> > > > > > > >     >> to do?
>>> > > > > > > >     >> >>> > Has
>>> > > > > > > >     >> >>> > > > anybody filed a ticket with Infra?
>>> > > > > > > >     >> >>> > > >
>>> > > > > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja
>>> > Pullela <
>>> > > > > > > >     >> >>> > > > raja.pullela@accelerite.com>
>>> > > > > > > >     >> >>> > > > wrote:
>>> > > > > > > >     >> >>> > > >
>>> > > > > > > >     >> >>> > > > > Hi will,
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > I believe, we didn’t get to close
>>> > > > > > > > ‘getting a
>>> > > > > mirror
>>> > > > > > on
>>> > > > > > > > Apache’
>>> > > > > > > >     >> >>> > because
>>> > > > > > > >     >> >>> > > we
>>> > > > > > > >     >> >>> > > > > needed someone on the Apache Infra side
>>> > > > > > > > to
>>> > > close
>>> > > > > > this.
>>> > > > > > > > BTW,
>>> > > > > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux
>>> > > > > > > > manages
>>> > > > this?)
>>> > > > > > has
>>> > > > > > > >     >> >>> > > > > all/most of
>>> > > > > > > >     >> >>> > the
>>> > > > > > > >     >> >>> > > > > content.  Once we can close on the
> Apache
>>> > > mirror
>>> > > > > for
>>> > > > > > > > hosting
>>> > > > > > > >     >> >>> > > > > the
>>> > > > > > > >     >> >>> > > > content, I
>>> > > > > > > >     >> >>> > > > > can help assist getting the content
> there.
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > For now, we have replicated the
>>> > > > > download.cloud.com
>>> > > > > > > > content to
>>> > > > > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
>>> > > > > > > >     >> >>> > > > > Also, we are working on a set of
>>> > > steps/procedure
>>> > > > > to
>>> > > > > > > > help with
>>> > > > > > > >     >> >>> > > > > this change.  I will update everyone in
>>> > > > > > > > about
>>> > > a
>>> > > > > > week’s
>>> > > > > > > > time
>>> > > > > > > >     >> >>> > > > > on the
>>> > > > > > > >     >> >>> > details.
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > Best,
>>> > > > > > > >     >> >>> > > > > Raja Pullela
>>> > > > > > > >     >> >>> > > > > Engineering Team,
>>> > > > > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa
>>> > > > > > > > Clara,
>>> > > > CA,
>>> > > > > > > 95054
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "
>>> > > williamstevens@gmail.com
>>> > > > > on
>>> > > > > > > > behalf of
>>> > > > > > > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com
>>> > > > > > > > on
>>> > > > behalf
>>> > > > > > of
>>> > > > > > > >     >> >>> > wstevens@cloudops.com>
>>> > > > > > > >     >> >>> > > > > wrote:
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > unfortunately the template mirror
>>> > > > > > > > conversation
>>> > > > got
>>> > > > > > > > caught up
>>> > > > > > > >     >> >>> > > > > in
>>> > > > > > > >     >> >>> > details
>>> > > > > > > >     >> >>> > > > and
>>> > > > > > > >     >> >>> > > > > nobody took the lead on implementing a
>>> > > solution.
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > citrix has been pinging me every couple
>>> > > > > > > > months
>>> > > > to
>>> > > > > > say
>>> > > > > > > > 'dude,
>>> > > > > > > >     >> >>> > > > > we
>>> > > > > > > >     >> >>> need
>>> > > > > > > >     >> >>> > to
>>> > > > > > > >     >> >>> > > > > remove the dependency on
>>> > > > > > > > download.citrix.com
>>> > > ',
>>> > > > > but
>>> > > > > > i
>>> > > > > > > > have not
>>> > > > > > > >     >> >>> > > > > had
>>> > > > > > > >     >> >>> > the
>>> > > > > > > >     >> >>> > > > > cycles to get in and solve the problem.
>>> > > > > > > > the
>>> > > > > > shutdown
>>> > > > > > > > of that
>>> > > > > > > >     >> >>> > > > > is
>>> > > > > > > >     >> >>> > > imminent
>>> > > > > > > >     >> >>> > > > > right now, so we need to solve it asap.
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > *Will STEVENS*
>>> > > > > > > >     >> >>> > > > > Lead Developer
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul
>>> > Angus <
>>> > > > > > > >     >> >>> > paul.angus@shapeblue.com
>>> > > > > > > >     >> >>> > > >
>>> > > > > > > >     >> >>> > > > > wrote:
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > > Hi Nathan,
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > Ideally, if you put the template
>>> > > > > > > > location in
>>> > > > (or
>>> > > > > > use
>>> > > > > > > a
>>> > > > > > > >     >> >>> > > > > > template
>>> > > > > > > >     >> >>> > > defined
>>> > > > > > > >     >> >>> > > > > > in)  test_data.py then the actual
>>> > > > > > > > location
>>> > > can
>>> > > > > be
>>> > > > > > > >     >> >>> > > > > > overridden by
>>> > > > > > > >     >> >>> > > anyone
>>> > > > > > > >     >> >>> > > > > > testing.
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > For Trillian, we've copied all of the
>>> > > > templates
>>> > > > > > that
>>> > > > > > > > people
>>> > > > > > > >     >> >>> > > > > > have
>>> > > > > > > >     >> >>> > > define
>>> > > > > > > >     >> >>> > > > > to
>>> > > > > > > >     >> >>> > > > > > a local repo and then replace the
URLs
>>> > > > > > > > in test_data.py to
>>> > > > > > > >     >> >>> > > > > > reduce
>>> > > > > > > >     >> >>> > > > > bandwidth
>>> > > > > > > >     >> >>> > > > > > use and download times.
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > Ie:
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >             "bootableIso":
>>> > > > > > > >     >> >>> > > > > >                 {
>>> > > > > > > >     >> >>> > > > > >                     "displaytext":
> "Test
>>> > > > > Bootable
>>> > > > > > > > ISO",
>>> > > > > > > >     >> >>> > > > > >                     "name":
"testISO",
>>> > > > > > > >     >> >>> > > > > >                     "bootable": True,
>>> > > > > > > >     >> >>> > > > > >                     "ispublic":
False,
>>> > > > > > > >     >> >>> > > > > >                     "url": "{{
>>> > > > > > > marvin_images_location
>>> > > > > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
>>> > > > > > > >     >> >>> > > > > >                     "ostype": 'Other
>>> Linux
>>> > > > > > > (64-bit)',
>>> > > > > > > >     >> >>> > > > > >                     "mode":
>>> > 'HTTP_DOWNLOAD'
>>> > > > > > > >     >> >>> > > > > >         },
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > I thought that we had come up with a
>>> > > solution
>>> > > > > for
>>> > > > > > > >     >> >>> > download.cloud.com
>>> > > > > > > >     >> >>> > > ,
>>> > > > > > > >     >> >>> > > > by
>>> > > > > > > >     >> >>> > > > > > having a mirrorlist hosted in
> Community
>>> > > Apache
>>> > > > > > > > 'space' with
>>> > > > > > > >     >> >>> anyone
>>> > > > > > > >     >> >>> > > able
>>> > > > > > > >     >> >>> > > > > to
>>> > > > > > > >     >> >>> > > > > > out themselves forward as a mirror.
>>> > > > > > > >     >> >>> > > > > > But I must admit I lost track of
>>> > > > > > > > whether
>>> > > > anyone
>>> > > > > > made
>>> > > > > > > > the
>>> > > > > > > >     >> >>> requisite
>>> > > > > > > >     >> >>> > > > > changes
>>> > > > > > > >     >> >>> > > > > > in code....
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > Kind regards,
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > Paul Angus
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > paul.angus@shapeblue.com
>>> > > > > > > >     >> >>> > > > > > www.shapeblue.com
>>> > > > > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden,
> London
>>> > > WC2N
>>> > > > > > 4HSUK
>>> > > > > > > >     >> >>> > > > > > @shapeblue
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > -----Original Message-----
>>> > > > > > > >     >> >>> > > > > > From: williamstevens@gmail.com
>>> > > > > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
>>> > > > > > > >     >> >>> > On
>>> > > > > > > >     >> >>> > > > > > Behalf Of Will Stevens
>>> > > > > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
>>> > > > > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
>>> > > > > > > >     >> >>> > > > > > Subject: Re: Modern template hosting
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > this is a hard questions.  in
general,
>>> > > > > > > > we
>>> > > > should
>>> > > > > > be
>>> > > > > > > > setting
>>> > > > > > > >     >> >>> > > > > > up a
>>> > > > > > > >     >> >>> > > mirror
>>> > > > > > > >     >> >>> > > > > on
>>> > > > > > > >     >> >>> > > > > > some cloudstack/apache domain and
then
>>> > > mirror
>>> > > > to
>>> > > > > > > other
>>> > > > > > > >     >> >>> > > > > > provided
>>> > > > > > > >     >> >>> > > > > templates.
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > we MUST come up with a solution to
>>> > deprecate
>>> > > > > > > >     >> >>> > > > > > 'download.cloud.com
>>> > > > > > > >     >> >>> ',
>>> > > > > > > >     >> >>> > > > that
>>> > > > > > > >     >> >>> > > > > > is going to be going away any day
now.
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > i don't know the right way to solve
>>> > > > > > > > this to
>>> > > be
>>> > > > > > > > honest, but
>>> > > > > > > >     >> >>> > > > > > if you
>>> > > > > > > >     >> >>> > > have
>>> > > > > > > >     >> >>> > > > > > ideas, i am willing to help.
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > *Will STEVENS*
>>> > > > > > > >     >> >>> > > > > > Lead Developer
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM,
>>> > > > > > > > Nathan
>>> > > > > Johnson <
>>> > > > > > > >     >> >>> njohnson@ena.com
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> > > > > wrote:
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > > So not to re-open a can of worms,
> but
>>> > > > > > > > I’m
>>> > > > in a
>>> > > > > > > > situation
>>> > > > > > > >     >> >>> > > > > > > where
>>> > > > > > > >     >> >>> I
>>> > > > > > > >     >> >>> > > need
>>> > > > > > > >     >> >>> > > > > > > to come up with a Marvin component
>>> > > > > > > > test
>>> > > that
>>> > > > > > > > depends on a
>>> > > > > > > >     >> >>> > template
>>> > > > > > > >     >> >>> > > > > > > based on a kernel that’s relatively
>>> > > > > > > > new,
>>> > > > i.e.,
>>> > > > > > > > newer than
>>> > > > > > > >     >> >>> Centos
>>> > > > > > > >     >> >>> > > 5.3
>>> > > > > > > >     >> >>> > > > /
>>> > > > > > > >     >> >>> > > > > > Ubuntu 10.04 .
>>> > > > > > > >     >> >>> > > > > > > I see openvm.eu has a suitable
>>> > > > > > > > template
>>> > > > > (Ubuntu
>>> > > > > > > > 16.0.4
>>> > > > > > > >     >> >>> > > > > > > for
>>> > > > > > > >     >> >>> KVM),
>>> > > > > > > >     >> >>> > > but
>>> > > > > > > >     >> >>> > > > > > > from looking at the thread
> "Migrating
>>> > > > > CloudStack
>>> > > > > > > > content
>>> > > > > > > >     >> >>> > > > > > > from download.cloud.com” it looks
>>> > > > > > > > like
>>> > > > there
>>> > > > > is
>>> > > > > > > >     >> >>> > > > > > > resistance to using
>>> > > > > > > >     >> >>> > > this
>>> > > > > > > >     >> >>> > > > at
>>> > > > > > > >     >> >>> > > > > > > least for hosting system vm
> templates
>>> > > > > > > > over
>>> > > > > > > concerns
>>> > > > > > > > of
>>> > > > > > > >     >> >>> > neutrality.
>>> > > > > > > >     >> >>> > > > > > > Would this be suitable for a
>>> > > > > > > > component
>>> > > test?
>>> > > > > If
>>> > > > > > > > not,
>>> > > > > > > >     >> >>> > > > > > > what is a
>>> > > > > > > >     >> >>> > > > > “blessed”
>>> > > > > > > >     >> >>> > > > > > > template location?
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > > Thanks in advance!
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > > Nathan Johnson
>>> > > > > > > >     >> >>> > > > > > > R&D Engineer
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12
>>> > > > Nashville,
>>> > > > > TN
>>> > > > > > > > 37211
>>> > > > > > > >     >> >>> > > > > > > General Office: 615-312-6000
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > > website | blog | support
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > DISCLAIMER
>>> > > > > > > >     >> >>> > > > > ==========
>>> > > > > > > >     >> >>> > > > > This e-mail may contain privileged and
>>> > > > > confidential
>>> > > > > > > >     >> >>> > > > > information
>>> > > > > > > >     >> >>> which
>>> > > > > > > >     >> >>> > > is
>>> > > > > > > >     >> >>> > > > > the property of Accelerite, a
Persistent
>>> > > Systems
>>> > > > > > > > business. It
>>> > > > > > > >     >> >>> > > > > is
>>> > > > > > > >     >> >>> > > intended
>>> > > > > > > >     >> >>> > > > > only for the use of the individual or
>>> > > > > > > > entity
>>> > > to
>>> > > > > > which
>>> > > > > > > > it is
>>> > > > > > > >     >> >>> > addressed.
>>> > > > > > > >     >> >>> > > If
>>> > > > > > > >     >> >>> > > > > you are not the intended recipient, you
>>> > > > > > > > are
>>> > > not
>>> > > > > > > > authorized to
>>> > > > > > > >     >> >>> > > > > read,
>>> > > > > > > >     >> >>> > > > retain,
>>> > > > > > > >     >> >>> > > > > copy, print, distribute or use this
>>> message.
>>> > > If
>>> > > > > you
>>> > > > > > > have
>>> > > > > > > >     >> >>> > > > > received
>>> > > > > > > >     >> >>> > this
>>> > > > > > > >     >> >>> > > > > communication in error, please notify
> the
>>> > > sender
>>> > > > > and
>>> > > > > > > > delete
>>> > > > > > > >     >> >>> > > > > all
>>> > > > > > > >     >> >>> > copies
>>> > > > > > > >     >> >>> > > of
>>> > > > > > > >     >> >>> > > > > this message. Accelerite, a Persistent
>>> > > > > > > > Systems
>>> > > > > > > business
>>> > > > > > > > does
>>> > > > > > > >     >> >>> > > > > not
>>> > > > > > > >     >> >>> > accept
>>> > > > > > > >     >> >>> > > > any
>>> > > > > > > >     >> >>> > > > > liability for virus infected mails.
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > >
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> >
>>> > > > > > > >     >> >>>
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >abhinandan.prateek@shapeblue.com
>>> > > > > > > >     >> >www.shapeblue.com
>>> > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N
4HSUK
>>> > > > > @shapeblue
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >
>>> > > > > > > >     >>
>>> > > > > > > >     >> abhinandan.prateek@shapeblue.com
>>> > > > > > > >     >> www.shapeblue.com
>>> > > > > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>> > > > > > > >     >> @shapeblue
>>> > > > > > > >     >>
>>> > > > > > > >     >>
>>> > > > > > > >     >>
>>> > > > > > > >     >>
>>> > > > > > > >     >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > daan.hoogland@shapeblue.com
>>> > > > > > > > www.shapeblue.com
>>> > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>> > > > > > > > @shapeblue
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > --
>>> > > > > > > Rafael Weingärtner
>>> > > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > --
>>> > > > > > Rafael Weingärtner
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>

Re: Modern template hosting

Posted by Erik Weber <te...@gmail.com>.

It'll most likely only be an issue in the case where a mirror goes
down without any immediate chance of getting back up, right?
Could we check with ASF Infra beforehand if such case is OK to warrant
an urgent Jira ticket to resolve - should it happen?

We can document all the mirrors and provide the howtos for manually
downloading and seeding the image as a backup.

Do we have any traffic statistics for the current solution, or is that
not available at S3?

-- 
Erik

On Sat, Mar 4, 2017 at 12:28 AM, Will Stevens <wi...@gmail.com> wrote:
> Yes Erik, I agree with you. The only thing that could add complexity in
> this is the fact that we don't control the domain, the ASF does (from what
> I understand).  Do we expect this to be managed by ASF infra or would there
> be another way? Ideally the domain is community controlled so we can adapt
> in case of change without long delays.
>
> I am 100% in agreement that this is the shortest path and likely to be good
> enough.
>
> On Mar 3, 2017 6:19 PM, "Erik Weber" <te...@gmail.com> wrote:
>
> IMHO; implement it as simple as possible, use DNS RR and assume/expect
> mirror hosts to use the same path.
>
>
> Yes I know; DNS RR isn't bulletproof, but as already mentioned in this
> thread - mirrors rarely go down and they aren't used all that often
> (primarily seeding).
> We can keep a low TTL on the record so that we're able to remove a
> mirror that goes down for a significant period of time (more than
> hours).
>
> And yes I know, some operators might find it a burden having to set up
> a new vhost with another document root, but tbh if they don't care
> enough to do it we'll manage without that mirror.
>
> --
> Erik
>
> On Sat, Mar 4, 2017 at 12:09 AM, Will Stevens <wi...@gmail.com>
> wrote:
>> I agree with Paul. Look at the list of things they have to
>> learn/master/care about. We don't want to add to that list, we want to
>> remove from that list. Make the project/product more accessible. Reduce
> the
>> barrier to entry...
>>
>> On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:
>>
>>> Seriously?
>>> apt-get install apache2
>>>
>>> To install CloudStack, they need to know
>>> - DB installation, perhaps some SQL
>>> - VLAN configuration on their switches
>>> - Ins-and-outs between port forwarding, static ips,
>>> - NFS
>>> - package management
>>> - VHDs, qcow2. vmdk
>>> - hyperviosrs
>>> - and on and on.
>>>
>>> If they can figure all that out and not figure out a web server, they can
>>> always come to the mailing list.
>>>
>>>
>>>
>>>
>>> On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <pa...@shapeblue.com>
>>> wrote:
>>>
>>> > The issue is not with supporting highly experienced cloud operators. It
>>> is
>>> > an issue for new users and other 'relatively' inexperienced operators.
>>> >
>>> > I have helped enough newbies and cloud operators who have been running
>>> > their cloud for a while to know that the barriers to entry are too high
>>> as
>>> > they are. And telling anyone that they need to create a web server so
>>> that
>>> > they can add their initial template to get started or in order to
> create
>>> a
>>> > new zone, just isn't going to fly.
>>> >
>>> > I'm a huge advocate of the 'download.cloudstack.org' endpoint which the
>>> > community can add/remove mirrors to or from for system VMs or built-in
>>> > templates.  Can the same system be used for binary repos ? although
> there
>>> > is an added complication of redist vs no-redist there...
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > paul.angus@shapeblue.com
>>> > www.shapeblue.com
>>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>> > @shapeblue
>>> >
>>> >
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: Chiradeep Vittal [mailto:chiradeepv@gmail.com]
>>> > Sent: 03 March 2017 18:28
>>> > To: dev <de...@cloudstack.apache.org>
>>> > Subject: Re: Modern template hosting
>>> >
>>> > I do feel like this is early optimization. Mirrors rarely fail. I'd
>>> expect
>>> > a single web server hosted on Apache Infra without any monitors to fail
>>> > more often than a mirror. We already expect Wido's systemvm repository
> to
>>> > be up all the time. And it has been. Similarly, I don't believe Nux's
>>> > repository has ever been down. And if Accelerite wants to host on S3,
>>> that
>>> > one is pretty solid as well.
>>> >
>>> > This is an infrequent operation in a cloud. After the cloud is
> installed,
>>> > the download servers are only needed for a new zone. If we trust the
> user
>>> > to run a cloud, surely he/she can run a web server to serve some
> built-in
>>> > templates. And if her cloud is successful, she needs to figure out how
> to
>>> > host her templates anyway and not rely on 3rd parties.
>>> >
>>> >
>>> > On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <ws...@cloudops.com>
>>> > wrote:
>>> >
>>> > > 1) If the legacy implementations do not support redirects, that does
>>> > > cause a problem.  A potential solution in that case is to have the
> web
>>> > > server actually proxy the download, but that is not ideal and I would
>>> > > like to avoid it if possible.  Thanks for bringing that up Chiradeep.
>>> > >
>>> > > 2) I think we need to have a single URL which people can target.
> Once
>>> > > they make the switch to the new URL, we want the implementation to be
>>> > > able to handle mirror failures without affecting the end client.  We
>>> > > want to avoid the situation where an ACS user will ever have to
> change
>>> > > this URL more than once.  Mirror failures SHOULD NOT affect the ACS
>>> > > users assuming there is still at least one mirror who can serve the
>>> > requested resource.
>>> > >
>>> > > These are obviously my personal opinions and others will probably
> have
>>> > > differing opinions.
>>> > >
>>> > > *Will STEVENS*
>>> > > Lead Developer
>>> > >
>>> > > <https://goo.gl/NYZ8KK>
>>> > >
>>> > > On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal
>>> > > <ch...@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > 1. If you are targeting legacy installations, they are not able to
>>> > > > follow redirects. The line of code that added this capability was
>>> > > > added on
>>> > > 11/16.
>>> > > > 2. If you trust the users to edit the database to change the URL,
>>> > > > you can trust them to change it to anything. Just document a known
>>> > > > list of good template locations.
>>> > > >
>>> > > > On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <wstevens@cloudops.com
>>
>>> > > > wrote:
>>> > > >
>>> > > > > So the main issue I see with this is the following.
>>> > > > >
>>> > > > > This implementation is designed to target legacy installations
>>> > > > > which
>>> > > will
>>> > > > > be affected when download.cloud.com disappears.  These people
> will
>>> > > need
>>> > > > to
>>> > > > > make a DB change to replace the 'download.cloud.com' with some
>>> > > > > other
>>> > > url
>>> > > > > (maybe 'download.cloudstack.org').  Once the DB has been updated,
>>> > > > > we
>>> > > can
>>> > > > > not expect anything else of the client, they need to be able to
>>> > > > > just continue operation as they were without any need to rebuild
>>> > > > > or upgrade their ACS.
>>> > > > >
>>> > > > > If we try to force the decision to the client, then we break the
>>> > > > > legacy implementations.  Since the SSVM is likely going to be the
>>> > > > > client in
>>> > > some
>>> > > > > cases, and since it does not already have the logic to handle the
>>> > > > > 300 approach correctly, I feel like it is not a viable solution
>>> > > > > for the
>>> > > > legacy
>>> > > > > deployments.
>>> > > > >
>>> > > > > In addition to that.  I don't think the client is in any better
>>> > > position
>>> > > > to
>>> > > > > make the mirror decision than I am.  I am likely able to make a
>>> > > > > more 'educated' decision than the client would be able to because
>>> > > > > I can do different tests on the endpoint(s) before making a final
>>> > decision.
>>> > > > >
>>> > > > > Does this make sense or am I missing something obvious?
>>> > > > >
>>> > > > > *Will STEVENS*
>>> > > > > Lead Developer
>>> > > > >
>>> > > > > <https://goo.gl/NYZ8KK>
>>> > > > >
>>> > > > > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
>>> > > > > rafaelweingartner@gmail.com> wrote:
>>> > > > >
>>> > > > > > I think I understood you, but I did not understand you.
>>> > > > > >
>>> > > > > > Let me see if I can create an example to illustrate how we
> could
>>> > > > > > do
>>> > > > using
>>> > > > > > the HTTP 300 code.
>>> > > > > >
>>> > > > > > The HTTP 300 code indicates clients that he/she/it has multiple
>>> > > choices
>>> > > > > to
>>> > > > > > access the requested resource. If the request is a HEAD, this
> is
>>> > > > > > the
>>> > > > only
>>> > > > > > thing that the server returns. If the request if a GET, then
> the
>>> > > server
>>> > > > > > returns the choices for the client. The response type is
> defined
>>> > > > > > on
>>> > > the
>>> > > > > > server based on the Content-Type, user agent and others.
>>> > > > > >
>>> > > > > > Let’s say the server receives a GET request and informs the
>>> > > > content-type
>>> > > > > as
>>> > > > > > JSON (application/JSON). Your application could get the mirrors
>>> > > > > > list,
>>> > > > do
>>> > > > > > the appending process for the requested resource path, and then
>>> > > return
>>> > > > a
>>> > > > > > JSON array with the possible mirror the client can use. Then,
>>> > > > > > the
>>> > > > client
>>> > > > > > gets this list and does the processing required to select a
>>> mirror.
>>> > > > > >
>>> > > > > > Now let’s say that a human access the link using a browser.
>>> > > > > > Then, the server should receive something like (text/plain) as
>>> > > > > > the
>>> > > content-type;
>>> > > > I
>>> > > > > > think depending on the browser this field may differ a little
>>> > > > > > (it
>>> > > would
>>> > > > > > require some checking). Anyways, the server detects that it is
> a
>>> > > > “human”
>>> > > > > > requesting the resource, then we could serve an HTML page with
> a
>>> > > > > Javascript
>>> > > > > > that uses the mirror list. This Javascript could do some
>>> > > > > > checking and choose the best mirror for that specific.
>>> > > > > >
>>> > > > > > Why do I think this approach is interesting?
>>> > > > > > The application you developed would be used only to retrieve
>>> > > > > > valid
>>> > > and
>>> > > > > > trusted sources of system VM images (a trusted repository for
>>> > > > > > mirrors
>>> > > > > that
>>> > > > > > we as a community vouch for).  Also, we shift the decision
>>> > > > > > process regarding mirrors from the server to the client. Then,
>>> > > > > > it is up to
>>> > > > > clients
>>> > > > > > to select mirrors, and not up to us (or some of our
>>> > implementations).
>>> > > > > >
>>> > > > > > Did this help?
>>> > > > > > BTW: I am not saying I am against the way you proposed, which
>>> > > > > > would
>>> > > > work
>>> > > > > > fine. It is merely a suggestion using a different perspective.
>>> > > > > >
>>> > > > > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
>>> > > > williamstevens@gmail.com>
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > > @rafael: in general, I don't think the client should ever be
>>> > > making a
>>> > > > > > > choice about a mirror. We have to assume we are working with
> a
>>> > > > scripted
>>> > > > > > > application and anything hard coded in that implementation on
>>> > > > > > > the
>>> > > > > client
>>> > > > > > > side is a risk.
>>> > > > > > >
>>> > > > > > > I may not be understanding the use of the 300 approach, so I
>>> > > > > > > will
>>> > > > > > research
>>> > > > > > > it to see if I can make it fit.
>>> > > > > > >
>>> > > > > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
>>> > > > > > rafaelweingartner@gmail.com>
>>> > > > > > > wrote:
>>> > > > > > >
>>> > > > > > > Will great job.
>>> > > > > > >
>>> > > > > > > I had the same doubt as Daan.
>>> > > > > > >
>>> > > > > > > Have you considered using HTTP 300 working mode? Then, we
>>> > > > > > > could let
>>> > > > the
>>> > > > > > > client decide which mirror is the best(closest?). Section
>>> > “10.3.1”
>>> > > of
>>> > > > > [1]
>>> > > > > > > talks about it; this would put the pressure on deciding upon
> a
>>> > > mirror
>>> > > > > on
>>> > > > > > > the client side, but I think it would be fair.
>>> > > > > > >
>>> > > > > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
>>> > > > > > >
>>> > > > > > > PS: You really like very short variables names!
>>> > > > > > >
>>> > > > > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
>>> > > > > > daan.hoogland@shapeblue.com
>>> > > > > > > >
>>> > > > > > > wrote:
>>> > > > > > >
>>> > > > > > > > Nice little thing Will,
>>> > > > > > > >
>>> > > > > > > > One question: if I read the code correctly it ‘go’es and
>>> > > > > > > > tries
>>> > > all
>>> > > > > > > mirrors
>>> > > > > > > > at once and whichever responses first is redirected to the
>>> > > client.
>>> > > > > This
>>> > > > > > > > might well be the same every time. This might be the one
>>> > > > > > > > closest
>>> > > to
>>> > > > > the
>>> > > > > > > > mirror302 and not to the client. Is that correct or did I
>>> > > > > > > > miss a
>>> > > > > > weighing
>>> > > > > > > > algorithm hidden in there?
>>> > > > > > > >
>>> > > > > > > > Good coding,
>>> > > > > > > >
>>> > > > > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
>>> > > > > > > >
>>> > > > > > > >     Hey All,
>>> > > > > > > >     Please review this repo and tell me what you think:
>>> > > > > > > >     https://github.com/swill/mirror302
>>> > > > > > > >
>>> > > > > > > >     Let me know if you have questions or you would like me
>>> > > > > > > > to
>>> > > make
>>> > > > > > > changes
>>> > > > > > > >     before I bring this topic up with ASF Infra and open a
>>> > > > > > > > ticket
>>> > > > to
>>> > > > > > get
>>> > > > > > > > this
>>> > > > > > > >     implemented.
>>> > > > > > > >
>>> > > > > > > >     Cheers,
>>> > > > > > > >
>>> > > > > > > >     *Will Stevens*
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
>>> > > > > > wstevens@cloudops.com
>>> > > > > > > >
>>> > > > > > > > wrote:
>>> > > > > > > >
>>> > > > > > > >     > I am building a short term solution right now.  I
>>> > > > > > > > hopefully
>>> > > > > will
>>> > > > > > > > have the
>>> > > > > > > >     > building blocks in place by the weekend so we can
>>> > > > > > > > start
>>> > > > working
>>> > > > > > > with
>>> > > > > > > > Infra
>>> > > > > > > >     > to get it in place.
>>> > > > > > > >     >
>>> > > > > > > >     > We will have a web server which we can point
> something
>>> > like
>>> > > > > > > >     > downloads.cloudstack.org (or whatever url) at.  Then
>>> > > > > > > > we
>>> > > will
>>> > > > > > have
>>> > > > > > > > an ASF
>>> > > > > > > >     > repo which tracks a mirror list and exposes a static
>>> > > > > > > > site
>>> > > to
>>> > > > be
>>> > > > > > > able
>>> > > > > > > > to
>>> > > > > > > >     > query the mirror list.
>>> > > > > > > >     >
>>> > > > > > > >     > The web server will receive a request and will do a
>>> > > > > > > > 302
>>> > > > > redirect
>>> > > > > > to
>>> > > > > > > > the
>>> > > > > > > >     > appropriate resource on one of the mirrors.
>>> > > > > > > >     >
>>> > > > > > > >     > This gets us started.  Legacy environments will have
>>> > > > > > > > to do
>>> > > a
>>> > > > DB
>>> > > > > > > > change to
>>> > > > > > > >     > change from 'download.cloud.com' to '
>>> > > > downloads.cloudstack.org'
>>> > > > > > (or
>>> > > > > > > >     > whatever), but otherwise it should be pretty simple.
>>> > > > > > > >     >
>>> > > > > > > >     > We can improve how we deliver templates going
> forward,
>>> > > > > > > > but
>>> > > > this
>>> > > > > > > > obviously
>>> > > > > > > >     > requires some discussion still.  I will try to buy us
>>> > > > > > > > some
>>> > > > time
>>> > > > > > > with
>>> > > > > > > > an
>>> > > > > > > >     > implementation which solves our problems today...
>>> > > > > > > >     >
>>> > > > > > > >     > *Will STEVENS*
>>> > > > > > > >     > Lead Developer
>>> > > > > > > >     >
>>> > > > > > > >     > <https://goo.gl/NYZ8KK>
>>> > > > > > > >     >
>>> > > > > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
>>> > > > > > > >     > abhinandan.prateek@shapeblue.com> wrote:
>>> > > > > > > >     >
>>> > > > > > > >     >>
>>> > > > > > > >     >> Initial seeding is a manual step and that is not
>>> > > > > > > > going to
>>> > > > > change
>>> > > > > > > in
>>> > > > > > > > near
>>> > > > > > > >     >> future. A handy list of official places from where
>>> > > > > > > > these
>>> > > > > > templates
>>> > > > > > > > can be
>>> > > > > > > >     >> downloaded won’t harm this part of the installation.
>>> > > > > > > > Just
>>> > > > > allow
>>> > > > > > > > anyone
>>> > > > > > > >     >> credible who follows democratic process to publish
>>> > > > > > > > their
>>> > > > > > templates
>>> > > > > > > > with
>>> > > > > > > >     >> apache’s blessings.
>>> > > > > > > >     >>
>>> > > > > > > >     >>
>>> > > > > > > >     >> Coming to urls hard coded in db, the popular
>>> > > > > > > > suggestion is
>>> > > > to
>>> > > > > > > point
>>> > > > > > > > them
>>> > > > > > > >     >> to mirrors. This looks good but this is going to
> take
>>> > > > > > > > some
>>> > > > > > effort
>>> > > > > > > > that has
>>> > > > > > > >     >> not materialised yet. Once this change is made it
>>> > > > > > > > will be
>>> > > > > going
>>> > > > > > in
>>> > > > > > > > some
>>> > > > > > > >     >> future release like 4.11; the problem still remains
>>> > > > > > > > for
>>> > > the
>>> > > > > > > > releases that
>>> > > > > > > >     >> are already out there. In case access to
>>> > > download.cloud.com
>>> > > > > is
>>> > > > > > > > dropped
>>> > > > > > > >     >> at some point we will have to document the procedure
>>> > > > > > > > such
>>> > > > that
>>> > > > > > > > people know
>>> > > > > > > >     >> how to make these stable releases work. That is
> where
>>> > > > > > > > the
>>> > > > > > > procedure
>>> > > > > > > >     >> documented here https://shankerbalan.net/blog/
>>> > > > > > > >     >> seed-cloudstack-templates-offline/ should be
>>> > > > > > > > officially
>>> > > > > adapted
>>> > > > > > > > with a
>>> > > > > > > >     >> marketplace to pick up templates of choosing. Maybe
>>> > > > > > > > we can
>>> > > > > have
>>> > > > > > a
>>> > > > > > > > pre-setup
>>> > > > > > > >     >> script to make choices instead of written wiki. Note
>>> > > > > > > > that
>>> > > > this
>>> > > > > > > will
>>> > > > > > > > be
>>> > > > > > > >     >> anyway needed by the older releases.
>>> > > > > > > >     >>
>>> > > > > > > >     >>
>>> > > > > > > >     >>
>>> > > > > > > >     >>
>>> > > > > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
>>> > > > paul.angus@shapeblue.com>
>>> > > > > > > > wrote:
>>> > > > > > > >     >>
>>> > > > > > > >     >> >-1
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >We are trying to get MORE people to use CloudStack.
>>> > > > > > > > And
>>> > > > > > raising
>>> > > > > > > > the
>>> > > > > > > >     >> barrier to entry does nothing to help that.
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >Kind regards,
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >Paul Angus
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >paul.angus@shapeblue.com
>>> > > > > > > >     >> >www.shapeblue.com
>>> > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>> > > > > > > >     >> >@shapeblue
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >-----Original Message-----
>>> > > > > > > >     >> >From: Abhinandan Prateek [mailto:
> abhinandan.prateek@
>>> > > > > > > shapeblue.com]
>>> > > > > > > >     >> >Sent: 01 March 2017 09:39
>>> > > > > > > >     >> >To: dev@cloudstack.apache.org
>>> > > > > > > >     >> >Subject: Re: Modern template hosting
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >+1
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >After seeing this discussion running in circles
>>> > > > > > > > several
>>> > > > > times,
>>> > > > > > I
>>> > > > > > > > think
>>> > > > > > > >     >> we should at least get started with the simplest
>>> option.
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
>>> > > > > > chiradeepv@gmail.com>
>>> > > > > > > > wrote:
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >>The alternative foolproof way is to simply not
>>> > > > > > > > provide
>>> > > the
>>> > > > > > > > automatic
>>> > > > > > > >     >> >>download.
>>> > > > > > > >     >> >>
>>> > > > > > > >     >> >>Just document the mirror list URL where the
>>> > > > > > > > templates
>>> > > can
>>> > > > be
>>> > > > > > > > found. I
>>> > > > > > > >     >> >>don't know why people reject this option.
>>> > > > > > > >     >> >>
>>> > > > > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
>>> > > > > > > templates-offline/
>>> > > > > > > >     >> >>
>>> > > > > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
>>> > > > > > > >     >> >><wi...@gmail.com>
>>> > > > > > > >     >> >>wrote:
>>> > > > > > > >     >> >>
>>> > > > > > > >     >> >>> OK. Thanks for the heads up.
>>> > > > > > > >     >> >>>
>>> > > > > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
>>> > > > > > > > chiradeepv@gmail.com>
>>> > > > > > > >     >> wrote:
>>> > > > > > > >     >> >>>
>>> > > > > > > >     >> >>> > Sounds workable. The downloader code in the
>>> > > > > > > > SSVM
>>> > > won't
>>> > > > > > > follow
>>> > > > > > > >     >> >>> > redirects I think.
>>> > > > > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
>>> > > > > > > 5511065fc20787619d
>>> > > > > > > > 9cd0444
>>> > > > > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/
>>> > > storage/template/
>>> > > > > > > >     >> >>> > HttpTemplateDownloader.java#L93
>>> > > > > > > >     >> >>> > https://goo.gl/dSi0r5
>>> > > > > > > >     >> >>> >  Might need to add
>>> > > > > > > >     >> >>> > client.setRedirectStrategy(new
>>> > > LaxRedirectStrategy());
>>> > > > > > > >     >> >>> >
>>> > > > > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
>>> > > > > > > >     >> >>> > <ws...@cloudops.com>
>>> > > > > > > >     >> >>> > wrote:
>>> > > > > > > >     >> >>> >
>>> > > > > > > >     >> >>> > > We haven't opened a ticket yet because we
>>> > > > > > > > don't
>>> > > > have a
>>> > > > > > > > strategy
>>> > > > > > > >     >> yet.
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> > > What do you guys think of this:
>>> > > > > > > >     >> >>> > > - We setup a new github repo in the 'apache'
>>> > > > > > > > org
>>> > > > which
>>> > > > > > > > consists
>>> > > > > > > >     >> >>> > > of a
>>> > > > > > > >     >> >>> > single
>>> > > > > > > >     >> >>> > > file with a list of active/supported
> mirrors.
>>> > > > > > > >     >> >>> > > - I write a small web server, distributed as
>>> > > > > > > > a
>>> > > > binary,
>>> > > > > > > > which can
>>> > > > > > > >     >> >>> > > be
>>> > > > > > > >     >> >>> > hosted
>>> > > > > > > >     >> >>> > > by ASF Infra.  This web server will query
> the
>>> > > > current
>>> > > > > > list
>>> > > > > > > > of
>>> > > > > > > >     >> >>> > > mirrors
>>> > > > > > > >     >> >>> and
>>> > > > > > > >     >> >>> > > will select one and then do a 302 redirect
> to
>>> > > > > > > > that
>>> > > > > > mirror.
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> > > The act of 'choosing' a mirror could be done
>>> > > > > > > > in a
>>> > > > > number
>>> > > > > > > of
>>> > > > > > > > ways.
>>> > > > > > > >     >> >>> > > - If we want to define an order, then it
>>> > > > > > > > could
>>> > > just
>>> > > > > try
>>> > > > > > > > from the
>>> > > > > > > >     >> >>> > > top of
>>> > > > > > > >     >> >>> > the
>>> > > > > > > >     >> >>> > > list and work its way down.  It would curl
>>> > > > > > > > the
>>> > > > target
>>> > > > > to
>>> > > > > > > > make
>>> > > > > > > >     >> >>> > > sure it
>>> > > > > > > >     >> >>> > gets
>>> > > > > > > >     >> >>> > > a 200 and if it does, it would do a 302
>>> > redirect.
>>> > > > > > > >     >> >>> > > - Or, if we want to distribute the load
>>> > > > > > > > across the
>>> > > > > > > mirrors,
>>> > > > > > > > we
>>> > > > > > > >     >> >>> > > could
>>> > > > > > > >     >> >>> pick
>>> > > > > > > >     >> >>> > > from the list randomly.  Again, doing a curl
>>> > > > > > > > to
>>> > > > verify
>>> > > > > > the
>>> > > > > > > > mirror
>>> > > > > > > >     >> >>> > > is up
>>> > > > > > > >     >> >>> > and
>>> > > > > > > >     >> >>> > > then doing a redirect.
>>> > > > > > > >     >> >>> > > - If we want to get fancy, we could do a
>>> > > > > > > > reverse
>>> > > IP
>>> > > > > > lookup
>>> > > > > > > > and
>>> > > > > > > >     >> >>> > > try to
>>> > > > > > > >     >> >>> > match
>>> > > > > > > >     >> >>> > > the requester with their closest
> geographical
>>> > > > mirror.
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> > > Thoughts?
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> > > *Will STEVENS*
>>> > > > > > > >     >> >>> > > Lead Developer
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep
>>> > > Vittal <
>>> > > > > > > >     >> >>> chiradeepv@gmail.com
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> > > wrote:
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> > > > What steps are needed to set up a mirror?
>>> > > > > > > > What
>>> > > > does
>>> > > > > > > Infra
>>> > > > > > > > need
>>> > > > > > > >     >> to do?
>>> > > > > > > >     >> >>> > Has
>>> > > > > > > >     >> >>> > > > anybody filed a ticket with Infra?
>>> > > > > > > >     >> >>> > > >
>>> > > > > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja
>>> > Pullela <
>>> > > > > > > >     >> >>> > > > raja.pullela@accelerite.com>
>>> > > > > > > >     >> >>> > > > wrote:
>>> > > > > > > >     >> >>> > > >
>>> > > > > > > >     >> >>> > > > > Hi will,
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > I believe, we didn’t get to close
>>> > > > > > > > ‘getting a
>>> > > > > mirror
>>> > > > > > on
>>> > > > > > > > Apache’
>>> > > > > > > >     >> >>> > because
>>> > > > > > > >     >> >>> > > we
>>> > > > > > > >     >> >>> > > > > needed someone on the Apache Infra side
>>> > > > > > > > to
>>> > > close
>>> > > > > > this.
>>> > > > > > > > BTW,
>>> > > > > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux
>>> > > > > > > > manages
>>> > > > this?)
>>> > > > > > has
>>> > > > > > > >     >> >>> > > > > all/most of
>>> > > > > > > >     >> >>> > the
>>> > > > > > > >     >> >>> > > > > content.  Once we can close on the
> Apache
>>> > > mirror
>>> > > > > for
>>> > > > > > > > hosting
>>> > > > > > > >     >> >>> > > > > the
>>> > > > > > > >     >> >>> > > > content, I
>>> > > > > > > >     >> >>> > > > > can help assist getting the content
> there.
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > For now, we have replicated the
>>> > > > > download.cloud.com
>>> > > > > > > > content to
>>> > > > > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
>>> > > > > > > >     >> >>> > > > > Also, we are working on a set of
>>> > > steps/procedure
>>> > > > > to
>>> > > > > > > > help with
>>> > > > > > > >     >> >>> > > > > this change.  I will update everyone in
>>> > > > > > > > about
>>> > > a
>>> > > > > > week’s
>>> > > > > > > > time
>>> > > > > > > >     >> >>> > > > > on the
>>> > > > > > > >     >> >>> > details.
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > Best,
>>> > > > > > > >     >> >>> > > > > Raja Pullela
>>> > > > > > > >     >> >>> > > > > Engineering Team,
>>> > > > > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa
>>> > > > > > > > Clara,
>>> > > > CA,
>>> > > > > > > 95054
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "
>>> > > williamstevens@gmail.com
>>> > > > > on
>>> > > > > > > > behalf of
>>> > > > > > > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com
>>> > > > > > > > on
>>> > > > behalf
>>> > > > > > of
>>> > > > > > > >     >> >>> > wstevens@cloudops.com>
>>> > > > > > > >     >> >>> > > > > wrote:
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > unfortunately the template mirror
>>> > > > > > > > conversation
>>> > > > got
>>> > > > > > > > caught up
>>> > > > > > > >     >> >>> > > > > in
>>> > > > > > > >     >> >>> > details
>>> > > > > > > >     >> >>> > > > and
>>> > > > > > > >     >> >>> > > > > nobody took the lead on implementing a
>>> > > solution.
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > citrix has been pinging me every couple
>>> > > > > > > > months
>>> > > > to
>>> > > > > > say
>>> > > > > > > > 'dude,
>>> > > > > > > >     >> >>> > > > > we
>>> > > > > > > >     >> >>> need
>>> > > > > > > >     >> >>> > to
>>> > > > > > > >     >> >>> > > > > remove the dependency on
>>> > > > > > > > download.citrix.com
>>> > > ',
>>> > > > > but
>>> > > > > > i
>>> > > > > > > > have not
>>> > > > > > > >     >> >>> > > > > had
>>> > > > > > > >     >> >>> > the
>>> > > > > > > >     >> >>> > > > > cycles to get in and solve the problem.
>>> > > > > > > > the
>>> > > > > > shutdown
>>> > > > > > > > of that
>>> > > > > > > >     >> >>> > > > > is
>>> > > > > > > >     >> >>> > > imminent
>>> > > > > > > >     >> >>> > > > > right now, so we need to solve it asap.
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > *Will STEVENS*
>>> > > > > > > >     >> >>> > > > > Lead Developer
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul
>>> > Angus <
>>> > > > > > > >     >> >>> > paul.angus@shapeblue.com
>>> > > > > > > >     >> >>> > > >
>>> > > > > > > >     >> >>> > > > > wrote:
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > > Hi Nathan,
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > Ideally, if you put the template
>>> > > > > > > > location in
>>> > > > (or
>>> > > > > > use
>>> > > > > > > a
>>> > > > > > > >     >> >>> > > > > > template
>>> > > > > > > >     >> >>> > > defined
>>> > > > > > > >     >> >>> > > > > > in)  test_data.py then the actual
>>> > > > > > > > location
>>> > > can
>>> > > > > be
>>> > > > > > > >     >> >>> > > > > > overridden by
>>> > > > > > > >     >> >>> > > anyone
>>> > > > > > > >     >> >>> > > > > > testing.
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > For Trillian, we've copied all of the
>>> > > > templates
>>> > > > > > that
>>> > > > > > > > people
>>> > > > > > > >     >> >>> > > > > > have
>>> > > > > > > >     >> >>> > > define
>>> > > > > > > >     >> >>> > > > > to
>>> > > > > > > >     >> >>> > > > > > a local repo and then replace the URLs
>>> > > > > > > > in test_data.py to
>>> > > > > > > >     >> >>> > > > > > reduce
>>> > > > > > > >     >> >>> > > > > bandwidth
>>> > > > > > > >     >> >>> > > > > > use and download times.
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > Ie:
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >             "bootableIso":
>>> > > > > > > >     >> >>> > > > > >                 {
>>> > > > > > > >     >> >>> > > > > >                     "displaytext":
> "Test
>>> > > > > Bootable
>>> > > > > > > > ISO",
>>> > > > > > > >     >> >>> > > > > >                     "name": "testISO",
>>> > > > > > > >     >> >>> > > > > >                     "bootable": True,
>>> > > > > > > >     >> >>> > > > > >                     "ispublic": False,
>>> > > > > > > >     >> >>> > > > > >                     "url": "{{
>>> > > > > > > marvin_images_location
>>> > > > > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
>>> > > > > > > >     >> >>> > > > > >                     "ostype": 'Other
>>> Linux
>>> > > > > > > (64-bit)',
>>> > > > > > > >     >> >>> > > > > >                     "mode":
>>> > 'HTTP_DOWNLOAD'
>>> > > > > > > >     >> >>> > > > > >         },
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > I thought that we had come up with a
>>> > > solution
>>> > > > > for
>>> > > > > > > >     >> >>> > download.cloud.com
>>> > > > > > > >     >> >>> > > ,
>>> > > > > > > >     >> >>> > > > by
>>> > > > > > > >     >> >>> > > > > > having a mirrorlist hosted in
> Community
>>> > > Apache
>>> > > > > > > > 'space' with
>>> > > > > > > >     >> >>> anyone
>>> > > > > > > >     >> >>> > > able
>>> > > > > > > >     >> >>> > > > > to
>>> > > > > > > >     >> >>> > > > > > out themselves forward as a mirror.
>>> > > > > > > >     >> >>> > > > > > But I must admit I lost track of
>>> > > > > > > > whether
>>> > > > anyone
>>> > > > > > made
>>> > > > > > > > the
>>> > > > > > > >     >> >>> requisite
>>> > > > > > > >     >> >>> > > > > changes
>>> > > > > > > >     >> >>> > > > > > in code....
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > Kind regards,
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > Paul Angus
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > paul.angus@shapeblue.com
>>> > > > > > > >     >> >>> > > > > > www.shapeblue.com
>>> > > > > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden,
> London
>>> > > WC2N
>>> > > > > > 4HSUK
>>> > > > > > > >     >> >>> > > > > > @shapeblue
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > -----Original Message-----
>>> > > > > > > >     >> >>> > > > > > From: williamstevens@gmail.com
>>> > > > > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
>>> > > > > > > >     >> >>> > On
>>> > > > > > > >     >> >>> > > > > > Behalf Of Will Stevens
>>> > > > > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
>>> > > > > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
>>> > > > > > > >     >> >>> > > > > > Subject: Re: Modern template hosting
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > this is a hard questions.  in general,
>>> > > > > > > > we
>>> > > > should
>>> > > > > > be
>>> > > > > > > > setting
>>> > > > > > > >     >> >>> > > > > > up a
>>> > > > > > > >     >> >>> > > mirror
>>> > > > > > > >     >> >>> > > > > on
>>> > > > > > > >     >> >>> > > > > > some cloudstack/apache domain and then
>>> > > mirror
>>> > > > to
>>> > > > > > > other
>>> > > > > > > >     >> >>> > > > > > provided
>>> > > > > > > >     >> >>> > > > > templates.
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > we MUST come up with a solution to
>>> > deprecate
>>> > > > > > > >     >> >>> > > > > > 'download.cloud.com
>>> > > > > > > >     >> >>> ',
>>> > > > > > > >     >> >>> > > > that
>>> > > > > > > >     >> >>> > > > > > is going to be going away any day now.
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > i don't know the right way to solve
>>> > > > > > > > this to
>>> > > be
>>> > > > > > > > honest, but
>>> > > > > > > >     >> >>> > > > > > if you
>>> > > > > > > >     >> >>> > > have
>>> > > > > > > >     >> >>> > > > > > ideas, i am willing to help.
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > *Will STEVENS*
>>> > > > > > > >     >> >>> > > > > > Lead Developer
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM,
>>> > > > > > > > Nathan
>>> > > > > Johnson <
>>> > > > > > > >     >> >>> njohnson@ena.com
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> > > > > wrote:
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > > > > So not to re-open a can of worms,
> but
>>> > > > > > > > I’m
>>> > > > in a
>>> > > > > > > > situation
>>> > > > > > > >     >> >>> > > > > > > where
>>> > > > > > > >     >> >>> I
>>> > > > > > > >     >> >>> > > need
>>> > > > > > > >     >> >>> > > > > > > to come up with a Marvin component
>>> > > > > > > > test
>>> > > that
>>> > > > > > > > depends on a
>>> > > > > > > >     >> >>> > template
>>> > > > > > > >     >> >>> > > > > > > based on a kernel that’s relatively
>>> > > > > > > > new,
>>> > > > i.e.,
>>> > > > > > > > newer than
>>> > > > > > > >     >> >>> Centos
>>> > > > > > > >     >> >>> > > 5.3
>>> > > > > > > >     >> >>> > > > /
>>> > > > > > > >     >> >>> > > > > > Ubuntu 10.04 .
>>> > > > > > > >     >> >>> > > > > > > I see openvm.eu has a suitable
>>> > > > > > > > template
>>> > > > > (Ubuntu
>>> > > > > > > > 16.0.4
>>> > > > > > > >     >> >>> > > > > > > for
>>> > > > > > > >     >> >>> KVM),
>>> > > > > > > >     >> >>> > > but
>>> > > > > > > >     >> >>> > > > > > > from looking at the thread
> "Migrating
>>> > > > > CloudStack
>>> > > > > > > > content
>>> > > > > > > >     >> >>> > > > > > > from download.cloud.com” it looks
>>> > > > > > > > like
>>> > > > there
>>> > > > > is
>>> > > > > > > >     >> >>> > > > > > > resistance to using
>>> > > > > > > >     >> >>> > > this
>>> > > > > > > >     >> >>> > > > at
>>> > > > > > > >     >> >>> > > > > > > least for hosting system vm
> templates
>>> > > > > > > > over
>>> > > > > > > concerns
>>> > > > > > > > of
>>> > > > > > > >     >> >>> > neutrality.
>>> > > > > > > >     >> >>> > > > > > > Would this be suitable for a
>>> > > > > > > > component
>>> > > test?
>>> > > > > If
>>> > > > > > > > not,
>>> > > > > > > >     >> >>> > > > > > > what is a
>>> > > > > > > >     >> >>> > > > > “blessed”
>>> > > > > > > >     >> >>> > > > > > > template location?
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > > Thanks in advance!
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > > Nathan Johnson
>>> > > > > > > >     >> >>> > > > > > > R&D Engineer
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12
>>> > > > Nashville,
>>> > > > > TN
>>> > > > > > > > 37211
>>> > > > > > > >     >> >>> > > > > > > General Office: 615-312-6000
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > > website | blog | support
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > > >
>>> > > > > > > >     >> >>> > > > > >
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > > > DISCLAIMER
>>> > > > > > > >     >> >>> > > > > ==========
>>> > > > > > > >     >> >>> > > > > This e-mail may contain privileged and
>>> > > > > confidential
>>> > > > > > > >     >> >>> > > > > information
>>> > > > > > > >     >> >>> which
>>> > > > > > > >     >> >>> > > is
>>> > > > > > > >     >> >>> > > > > the property of Accelerite, a Persistent
>>> > > Systems
>>> > > > > > > > business. It
>>> > > > > > > >     >> >>> > > > > is
>>> > > > > > > >     >> >>> > > intended
>>> > > > > > > >     >> >>> > > > > only for the use of the individual or
>>> > > > > > > > entity
>>> > > to
>>> > > > > > which
>>> > > > > > > > it is
>>> > > > > > > >     >> >>> > addressed.
>>> > > > > > > >     >> >>> > > If
>>> > > > > > > >     >> >>> > > > > you are not the intended recipient, you
>>> > > > > > > > are
>>> > > not
>>> > > > > > > > authorized to
>>> > > > > > > >     >> >>> > > > > read,
>>> > > > > > > >     >> >>> > > > retain,
>>> > > > > > > >     >> >>> > > > > copy, print, distribute or use this
>>> message.
>>> > > If
>>> > > > > you
>>> > > > > > > have
>>> > > > > > > >     >> >>> > > > > received
>>> > > > > > > >     >> >>> > this
>>> > > > > > > >     >> >>> > > > > communication in error, please notify
> the
>>> > > sender
>>> > > > > and
>>> > > > > > > > delete
>>> > > > > > > >     >> >>> > > > > all
>>> > > > > > > >     >> >>> > copies
>>> > > > > > > >     >> >>> > > of
>>> > > > > > > >     >> >>> > > > > this message. Accelerite, a Persistent
>>> > > > > > > > Systems
>>> > > > > > > business
>>> > > > > > > > does
>>> > > > > > > >     >> >>> > > > > not
>>> > > > > > > >     >> >>> > accept
>>> > > > > > > >     >> >>> > > > any
>>> > > > > > > >     >> >>> > > > > liability for virus infected mails.
>>> > > > > > > >     >> >>> > > > >
>>> > > > > > > >     >> >>> > > >
>>> > > > > > > >     >> >>> > >
>>> > > > > > > >     >> >>> >
>>> > > > > > > >     >> >>>
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >abhinandan.prateek@shapeblue.com
>>> > > > > > > >     >> >www.shapeblue.com
>>> > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>> > > > > @shapeblue
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >
>>> > > > > > > >     >> >
>>> > > > > > > >     >>
>>> > > > > > > >     >> abhinandan.prateek@shapeblue.com
>>> > > > > > > >     >> www.shapeblue.com
>>> > > > > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>> > > > > > > >     >> @shapeblue
>>> > > > > > > >     >>
>>> > > > > > > >     >>
>>> > > > > > > >     >>
>>> > > > > > > >     >>
>>> > > > > > > >     >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > daan.hoogland@shapeblue.com
>>> > > > > > > > www.shapeblue.com
>>> > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>> > > > > > > > @shapeblue
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > --
>>> > > > > > > Rafael Weingärtner
>>> > > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > --
>>> > > > > > Rafael Weingärtner
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>

Re: Modern template hosting

Posted by Will Stevens <wi...@gmail.com>.

Yes Erik, I agree with you. The only thing that could add complexity in
this is the fact that we don't control the domain, the ASF does (from what
I understand).  Do we expect this to be managed by ASF infra or would there
be another way? Ideally the domain is community controlled so we can adapt
in case of change without long delays.

I am 100% in agreement that this is the shortest path and likely to be good
enough.

On Mar 3, 2017 6:19 PM, "Erik Weber" <te...@gmail.com> wrote:

IMHO; implement it as simple as possible, use DNS RR and assume/expect
mirror hosts to use the same path.


Yes I know; DNS RR isn't bulletproof, but as already mentioned in this
thread - mirrors rarely go down and they aren't used all that often
(primarily seeding).
We can keep a low TTL on the record so that we're able to remove a
mirror that goes down for a significant period of time (more than
hours).

And yes I know, some operators might find it a burden having to set up
a new vhost with another document root, but tbh if they don't care
enough to do it we'll manage without that mirror.

--
Erik

On Sat, Mar 4, 2017 at 12:09 AM, Will Stevens <wi...@gmail.com>
wrote:
> I agree with Paul. Look at the list of things they have to
> learn/master/care about. We don't want to add to that list, we want to
> remove from that list. Make the project/product more accessible. Reduce
the
> barrier to entry...
>
> On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:
>
>> Seriously?
>> apt-get install apache2
>>
>> To install CloudStack, they need to know
>> - DB installation, perhaps some SQL
>> - VLAN configuration on their switches
>> - Ins-and-outs between port forwarding, static ips,
>> - NFS
>> - package management
>> - VHDs, qcow2. vmdk
>> - hyperviosrs
>> - and on and on.
>>
>> If they can figure all that out and not figure out a web server, they can
>> always come to the mailing list.
>>
>>
>>
>>
>> On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <pa...@shapeblue.com>
>> wrote:
>>
>> > The issue is not with supporting highly experienced cloud operators. It
>> is
>> > an issue for new users and other 'relatively' inexperienced operators.
>> >
>> > I have helped enough newbies and cloud operators who have been running
>> > their cloud for a while to know that the barriers to entry are too high
>> as
>> > they are. And telling anyone that they need to create a web server so
>> that
>> > they can add their initial template to get started or in order to
create
>> a
>> > new zone, just isn't going to fly.
>> >
>> > I'm a huge advocate of the 'download.cloudstack.org' endpoint which the
>> > community can add/remove mirrors to or from for system VMs or built-in
>> > templates.  Can the same system be used for binary repos ? although
there
>> > is an added complication of redist vs no-redist there...
>> >
>> >
>> >
>> >
>> >
>> >
>> > paul.angus@shapeblue.com
>> > www.shapeblue.com
>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > @shapeblue
>> >
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: Chiradeep Vittal [mailto:chiradeepv@gmail.com]
>> > Sent: 03 March 2017 18:28
>> > To: dev <de...@cloudstack.apache.org>
>> > Subject: Re: Modern template hosting
>> >
>> > I do feel like this is early optimization. Mirrors rarely fail. I'd
>> expect
>> > a single web server hosted on Apache Infra without any monitors to fail
>> > more often than a mirror. We already expect Wido's systemvm repository
to
>> > be up all the time. And it has been. Similarly, I don't believe Nux's
>> > repository has ever been down. And if Accelerite wants to host on S3,
>> that
>> > one is pretty solid as well.
>> >
>> > This is an infrequent operation in a cloud. After the cloud is
installed,
>> > the download servers are only needed for a new zone. If we trust the
user
>> > to run a cloud, surely he/she can run a web server to serve some
built-in
>> > templates. And if her cloud is successful, she needs to figure out how
to
>> > host her templates anyway and not rely on 3rd parties.
>> >
>> >
>> > On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <ws...@cloudops.com>
>> > wrote:
>> >
>> > > 1) If the legacy implementations do not support redirects, that does
>> > > cause a problem.  A potential solution in that case is to have the
web
>> > > server actually proxy the download, but that is not ideal and I would
>> > > like to avoid it if possible.  Thanks for bringing that up Chiradeep.
>> > >
>> > > 2) I think we need to have a single URL which people can target.
Once
>> > > they make the switch to the new URL, we want the implementation to be
>> > > able to handle mirror failures without affecting the end client.  We
>> > > want to avoid the situation where an ACS user will ever have to
change
>> > > this URL more than once.  Mirror failures SHOULD NOT affect the ACS
>> > > users assuming there is still at least one mirror who can serve the
>> > requested resource.
>> > >
>> > > These are obviously my personal opinions and others will probably
have
>> > > differing opinions.
>> > >
>> > > *Will STEVENS*
>> > > Lead Developer
>> > >
>> > > <https://goo.gl/NYZ8KK>
>> > >
>> > > On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal
>> > > <ch...@gmail.com>
>> > > wrote:
>> > >
>> > > > 1. If you are targeting legacy installations, they are not able to
>> > > > follow redirects. The line of code that added this capability was
>> > > > added on
>> > > 11/16.
>> > > > 2. If you trust the users to edit the database to change the URL,
>> > > > you can trust them to change it to anything. Just document a known
>> > > > list of good template locations.
>> > > >
>> > > > On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <wstevens@cloudops.com
>
>> > > > wrote:
>> > > >
>> > > > > So the main issue I see with this is the following.
>> > > > >
>> > > > > This implementation is designed to target legacy installations
>> > > > > which
>> > > will
>> > > > > be affected when download.cloud.com disappears.  These people
will
>> > > need
>> > > > to
>> > > > > make a DB change to replace the 'download.cloud.com' with some
>> > > > > other
>> > > url
>> > > > > (maybe 'download.cloudstack.org').  Once the DB has been updated,
>> > > > > we
>> > > can
>> > > > > not expect anything else of the client, they need to be able to
>> > > > > just continue operation as they were without any need to rebuild
>> > > > > or upgrade their ACS.
>> > > > >
>> > > > > If we try to force the decision to the client, then we break the
>> > > > > legacy implementations.  Since the SSVM is likely going to be the
>> > > > > client in
>> > > some
>> > > > > cases, and since it does not already have the logic to handle the
>> > > > > 300 approach correctly, I feel like it is not a viable solution
>> > > > > for the
>> > > > legacy
>> > > > > deployments.
>> > > > >
>> > > > > In addition to that.  I don't think the client is in any better
>> > > position
>> > > > to
>> > > > > make the mirror decision than I am.  I am likely able to make a
>> > > > > more 'educated' decision than the client would be able to because
>> > > > > I can do different tests on the endpoint(s) before making a final
>> > decision.
>> > > > >
>> > > > > Does this make sense or am I missing something obvious?
>> > > > >
>> > > > > *Will STEVENS*
>> > > > > Lead Developer
>> > > > >
>> > > > > <https://goo.gl/NYZ8KK>
>> > > > >
>> > > > > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
>> > > > > rafaelweingartner@gmail.com> wrote:
>> > > > >
>> > > > > > I think I understood you, but I did not understand you.
>> > > > > >
>> > > > > > Let me see if I can create an example to illustrate how we
could
>> > > > > > do
>> > > > using
>> > > > > > the HTTP 300 code.
>> > > > > >
>> > > > > > The HTTP 300 code indicates clients that he/she/it has multiple
>> > > choices
>> > > > > to
>> > > > > > access the requested resource. If the request is a HEAD, this
is
>> > > > > > the
>> > > > only
>> > > > > > thing that the server returns. If the request if a GET, then
the
>> > > server
>> > > > > > returns the choices for the client. The response type is
defined
>> > > > > > on
>> > > the
>> > > > > > server based on the Content-Type, user agent and others.
>> > > > > >
>> > > > > > Let’s say the server receives a GET request and informs the
>> > > > content-type
>> > > > > as
>> > > > > > JSON (application/JSON). Your application could get the mirrors
>> > > > > > list,
>> > > > do
>> > > > > > the appending process for the requested resource path, and then
>> > > return
>> > > > a
>> > > > > > JSON array with the possible mirror the client can use. Then,
>> > > > > > the
>> > > > client
>> > > > > > gets this list and does the processing required to select a
>> mirror.
>> > > > > >
>> > > > > > Now let’s say that a human access the link using a browser.
>> > > > > > Then, the server should receive something like (text/plain) as
>> > > > > > the
>> > > content-type;
>> > > > I
>> > > > > > think depending on the browser this field may differ a little
>> > > > > > (it
>> > > would
>> > > > > > require some checking). Anyways, the server detects that it is
a
>> > > > “human”
>> > > > > > requesting the resource, then we could serve an HTML page with
a
>> > > > > Javascript
>> > > > > > that uses the mirror list. This Javascript could do some
>> > > > > > checking and choose the best mirror for that specific.
>> > > > > >
>> > > > > > Why do I think this approach is interesting?
>> > > > > > The application you developed would be used only to retrieve
>> > > > > > valid
>> > > and
>> > > > > > trusted sources of system VM images (a trusted repository for
>> > > > > > mirrors
>> > > > > that
>> > > > > > we as a community vouch for).  Also, we shift the decision
>> > > > > > process regarding mirrors from the server to the client. Then,
>> > > > > > it is up to
>> > > > > clients
>> > > > > > to select mirrors, and not up to us (or some of our
>> > implementations).
>> > > > > >
>> > > > > > Did this help?
>> > > > > > BTW: I am not saying I am against the way you proposed, which
>> > > > > > would
>> > > > work
>> > > > > > fine. It is merely a suggestion using a different perspective.
>> > > > > >
>> > > > > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
>> > > > williamstevens@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > @rafael: in general, I don't think the client should ever be
>> > > making a
>> > > > > > > choice about a mirror. We have to assume we are working with
a
>> > > > scripted
>> > > > > > > application and anything hard coded in that implementation on
>> > > > > > > the
>> > > > > client
>> > > > > > > side is a risk.
>> > > > > > >
>> > > > > > > I may not be understanding the use of the 300 approach, so I
>> > > > > > > will
>> > > > > > research
>> > > > > > > it to see if I can make it fit.
>> > > > > > >
>> > > > > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
>> > > > > > rafaelweingartner@gmail.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > Will great job.
>> > > > > > >
>> > > > > > > I had the same doubt as Daan.
>> > > > > > >
>> > > > > > > Have you considered using HTTP 300 working mode? Then, we
>> > > > > > > could let
>> > > > the
>> > > > > > > client decide which mirror is the best(closest?). Section
>> > “10.3.1”
>> > > of
>> > > > > [1]
>> > > > > > > talks about it; this would put the pressure on deciding upon
a
>> > > mirror
>> > > > > on
>> > > > > > > the client side, but I think it would be fair.
>> > > > > > >
>> > > > > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
>> > > > > > >
>> > > > > > > PS: You really like very short variables names!
>> > > > > > >
>> > > > > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
>> > > > > > daan.hoogland@shapeblue.com
>> > > > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Nice little thing Will,
>> > > > > > > >
>> > > > > > > > One question: if I read the code correctly it ‘go’es and
>> > > > > > > > tries
>> > > all
>> > > > > > > mirrors
>> > > > > > > > at once and whichever responses first is redirected to the
>> > > client.
>> > > > > This
>> > > > > > > > might well be the same every time. This might be the one
>> > > > > > > > closest
>> > > to
>> > > > > the
>> > > > > > > > mirror302 and not to the client. Is that correct or did I
>> > > > > > > > miss a
>> > > > > > weighing
>> > > > > > > > algorithm hidden in there?
>> > > > > > > >
>> > > > > > > > Good coding,
>> > > > > > > >
>> > > > > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
>> > > > > > > >
>> > > > > > > >     Hey All,
>> > > > > > > >     Please review this repo and tell me what you think:
>> > > > > > > >     https://github.com/swill/mirror302
>> > > > > > > >
>> > > > > > > >     Let me know if you have questions or you would like me
>> > > > > > > > to
>> > > make
>> > > > > > > changes
>> > > > > > > >     before I bring this topic up with ASF Infra and open a
>> > > > > > > > ticket
>> > > > to
>> > > > > > get
>> > > > > > > > this
>> > > > > > > >     implemented.
>> > > > > > > >
>> > > > > > > >     Cheers,
>> > > > > > > >
>> > > > > > > >     *Will Stevens*
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
>> > > > > > wstevens@cloudops.com
>> > > > > > > >
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > >     > I am building a short term solution right now.  I
>> > > > > > > > hopefully
>> > > > > will
>> > > > > > > > have the
>> > > > > > > >     > building blocks in place by the weekend so we can
>> > > > > > > > start
>> > > > working
>> > > > > > > with
>> > > > > > > > Infra
>> > > > > > > >     > to get it in place.
>> > > > > > > >     >
>> > > > > > > >     > We will have a web server which we can point
something
>> > like
>> > > > > > > >     > downloads.cloudstack.org (or whatever url) at.  Then
>> > > > > > > > we
>> > > will
>> > > > > > have
>> > > > > > > > an ASF
>> > > > > > > >     > repo which tracks a mirror list and exposes a static
>> > > > > > > > site
>> > > to
>> > > > be
>> > > > > > > able
>> > > > > > > > to
>> > > > > > > >     > query the mirror list.
>> > > > > > > >     >
>> > > > > > > >     > The web server will receive a request and will do a
>> > > > > > > > 302
>> > > > > redirect
>> > > > > > to
>> > > > > > > > the
>> > > > > > > >     > appropriate resource on one of the mirrors.
>> > > > > > > >     >
>> > > > > > > >     > This gets us started.  Legacy environments will have
>> > > > > > > > to do
>> > > a
>> > > > DB
>> > > > > > > > change to
>> > > > > > > >     > change from 'download.cloud.com' to '
>> > > > downloads.cloudstack.org'
>> > > > > > (or
>> > > > > > > >     > whatever), but otherwise it should be pretty simple.
>> > > > > > > >     >
>> > > > > > > >     > We can improve how we deliver templates going
forward,
>> > > > > > > > but
>> > > > this
>> > > > > > > > obviously
>> > > > > > > >     > requires some discussion still.  I will try to buy us
>> > > > > > > > some
>> > > > time
>> > > > > > > with
>> > > > > > > > an
>> > > > > > > >     > implementation which solves our problems today...
>> > > > > > > >     >
>> > > > > > > >     > *Will STEVENS*
>> > > > > > > >     > Lead Developer
>> > > > > > > >     >
>> > > > > > > >     > <https://goo.gl/NYZ8KK>
>> > > > > > > >     >
>> > > > > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
>> > > > > > > >     > abhinandan.prateek@shapeblue.com> wrote:
>> > > > > > > >     >
>> > > > > > > >     >>
>> > > > > > > >     >> Initial seeding is a manual step and that is not
>> > > > > > > > going to
>> > > > > change
>> > > > > > > in
>> > > > > > > > near
>> > > > > > > >     >> future. A handy list of official places from where
>> > > > > > > > these
>> > > > > > templates
>> > > > > > > > can be
>> > > > > > > >     >> downloaded won’t harm this part of the installation.
>> > > > > > > > Just
>> > > > > allow
>> > > > > > > > anyone
>> > > > > > > >     >> credible who follows democratic process to publish
>> > > > > > > > their
>> > > > > > templates
>> > > > > > > > with
>> > > > > > > >     >> apache’s blessings.
>> > > > > > > >     >>
>> > > > > > > >     >>
>> > > > > > > >     >> Coming to urls hard coded in db, the popular
>> > > > > > > > suggestion is
>> > > > to
>> > > > > > > point
>> > > > > > > > them
>> > > > > > > >     >> to mirrors. This looks good but this is going to
take
>> > > > > > > > some
>> > > > > > effort
>> > > > > > > > that has
>> > > > > > > >     >> not materialised yet. Once this change is made it
>> > > > > > > > will be
>> > > > > going
>> > > > > > in
>> > > > > > > > some
>> > > > > > > >     >> future release like 4.11; the problem still remains
>> > > > > > > > for
>> > > the
>> > > > > > > > releases that
>> > > > > > > >     >> are already out there. In case access to
>> > > download.cloud.com
>> > > > > is
>> > > > > > > > dropped
>> > > > > > > >     >> at some point we will have to document the procedure
>> > > > > > > > such
>> > > > that
>> > > > > > > > people know
>> > > > > > > >     >> how to make these stable releases work. That is
where
>> > > > > > > > the
>> > > > > > > procedure
>> > > > > > > >     >> documented here https://shankerbalan.net/blog/
>> > > > > > > >     >> seed-cloudstack-templates-offline/ should be
>> > > > > > > > officially
>> > > > > adapted
>> > > > > > > > with a
>> > > > > > > >     >> marketplace to pick up templates of choosing. Maybe
>> > > > > > > > we can
>> > > > > have
>> > > > > > a
>> > > > > > > > pre-setup
>> > > > > > > >     >> script to make choices instead of written wiki. Note
>> > > > > > > > that
>> > > > this
>> > > > > > > will
>> > > > > > > > be
>> > > > > > > >     >> anyway needed by the older releases.
>> > > > > > > >     >>
>> > > > > > > >     >>
>> > > > > > > >     >>
>> > > > > > > >     >>
>> > > > > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
>> > > > paul.angus@shapeblue.com>
>> > > > > > > > wrote:
>> > > > > > > >     >>
>> > > > > > > >     >> >-1
>> > > > > > > >     >> >
>> > > > > > > >     >> >We are trying to get MORE people to use CloudStack.
>> > > > > > > > And
>> > > > > > raising
>> > > > > > > > the
>> > > > > > > >     >> barrier to entry does nothing to help that.
>> > > > > > > >     >> >
>> > > > > > > >     >> >
>> > > > > > > >     >> >Kind regards,
>> > > > > > > >     >> >
>> > > > > > > >     >> >Paul Angus
>> > > > > > > >     >> >
>> > > > > > > >     >> >paul.angus@shapeblue.com
>> > > > > > > >     >> >www.shapeblue.com
>> > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > > > > > > >     >> >@shapeblue
>> > > > > > > >     >> >
>> > > > > > > >     >> >
>> > > > > > > >     >> >
>> > > > > > > >     >> >
>> > > > > > > >     >> >-----Original Message-----
>> > > > > > > >     >> >From: Abhinandan Prateek [mailto:
abhinandan.prateek@
>> > > > > > > shapeblue.com]
>> > > > > > > >     >> >Sent: 01 March 2017 09:39
>> > > > > > > >     >> >To: dev@cloudstack.apache.org
>> > > > > > > >     >> >Subject: Re: Modern template hosting
>> > > > > > > >     >> >
>> > > > > > > >     >> >+1
>> > > > > > > >     >> >
>> > > > > > > >     >> >After seeing this discussion running in circles
>> > > > > > > > several
>> > > > > times,
>> > > > > > I
>> > > > > > > > think
>> > > > > > > >     >> we should at least get started with the simplest
>> option.
>> > > > > > > >     >> >
>> > > > > > > >     >> >
>> > > > > > > >     >> >
>> > > > > > > >     >> >
>> > > > > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
>> > > > > > chiradeepv@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >     >> >
>> > > > > > > >     >> >>The alternative foolproof way is to simply not
>> > > > > > > > provide
>> > > the
>> > > > > > > > automatic
>> > > > > > > >     >> >>download.
>> > > > > > > >     >> >>
>> > > > > > > >     >> >>Just document the mirror list URL where the
>> > > > > > > > templates
>> > > can
>> > > > be
>> > > > > > > > found. I
>> > > > > > > >     >> >>don't know why people reject this option.
>> > > > > > > >     >> >>
>> > > > > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
>> > > > > > > templates-offline/
>> > > > > > > >     >> >>
>> > > > > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
>> > > > > > > >     >> >><wi...@gmail.com>
>> > > > > > > >     >> >>wrote:
>> > > > > > > >     >> >>
>> > > > > > > >     >> >>> OK. Thanks for the heads up.
>> > > > > > > >     >> >>>
>> > > > > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
>> > > > > > > > chiradeepv@gmail.com>
>> > > > > > > >     >> wrote:
>> > > > > > > >     >> >>>
>> > > > > > > >     >> >>> > Sounds workable. The downloader code in the
>> > > > > > > > SSVM
>> > > won't
>> > > > > > > follow
>> > > > > > > >     >> >>> > redirects I think.
>> > > > > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
>> > > > > > > 5511065fc20787619d
>> > > > > > > > 9cd0444
>> > > > > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/
>> > > storage/template/
>> > > > > > > >     >> >>> > HttpTemplateDownloader.java#L93
>> > > > > > > >     >> >>> > https://goo.gl/dSi0r5
>> > > > > > > >     >> >>> >  Might need to add
>> > > > > > > >     >> >>> > client.setRedirectStrategy(new
>> > > LaxRedirectStrategy());
>> > > > > > > >     >> >>> >
>> > > > > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
>> > > > > > > >     >> >>> > <ws...@cloudops.com>
>> > > > > > > >     >> >>> > wrote:
>> > > > > > > >     >> >>> >
>> > > > > > > >     >> >>> > > We haven't opened a ticket yet because we
>> > > > > > > > don't
>> > > > have a
>> > > > > > > > strategy
>> > > > > > > >     >> yet.
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> > > What do you guys think of this:
>> > > > > > > >     >> >>> > > - We setup a new github repo in the 'apache'
>> > > > > > > > org
>> > > > which
>> > > > > > > > consists
>> > > > > > > >     >> >>> > > of a
>> > > > > > > >     >> >>> > single
>> > > > > > > >     >> >>> > > file with a list of active/supported
mirrors.
>> > > > > > > >     >> >>> > > - I write a small web server, distributed as
>> > > > > > > > a
>> > > > binary,
>> > > > > > > > which can
>> > > > > > > >     >> >>> > > be
>> > > > > > > >     >> >>> > hosted
>> > > > > > > >     >> >>> > > by ASF Infra.  This web server will query
the
>> > > > current
>> > > > > > list
>> > > > > > > > of
>> > > > > > > >     >> >>> > > mirrors
>> > > > > > > >     >> >>> and
>> > > > > > > >     >> >>> > > will select one and then do a 302 redirect
to
>> > > > > > > > that
>> > > > > > mirror.
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> > > The act of 'choosing' a mirror could be done
>> > > > > > > > in a
>> > > > > number
>> > > > > > > of
>> > > > > > > > ways.
>> > > > > > > >     >> >>> > > - If we want to define an order, then it
>> > > > > > > > could
>> > > just
>> > > > > try
>> > > > > > > > from the
>> > > > > > > >     >> >>> > > top of
>> > > > > > > >     >> >>> > the
>> > > > > > > >     >> >>> > > list and work its way down.  It would curl
>> > > > > > > > the
>> > > > target
>> > > > > to
>> > > > > > > > make
>> > > > > > > >     >> >>> > > sure it
>> > > > > > > >     >> >>> > gets
>> > > > > > > >     >> >>> > > a 200 and if it does, it would do a 302
>> > redirect.
>> > > > > > > >     >> >>> > > - Or, if we want to distribute the load
>> > > > > > > > across the
>> > > > > > > mirrors,
>> > > > > > > > we
>> > > > > > > >     >> >>> > > could
>> > > > > > > >     >> >>> pick
>> > > > > > > >     >> >>> > > from the list randomly.  Again, doing a curl
>> > > > > > > > to
>> > > > verify
>> > > > > > the
>> > > > > > > > mirror
>> > > > > > > >     >> >>> > > is up
>> > > > > > > >     >> >>> > and
>> > > > > > > >     >> >>> > > then doing a redirect.
>> > > > > > > >     >> >>> > > - If we want to get fancy, we could do a
>> > > > > > > > reverse
>> > > IP
>> > > > > > lookup
>> > > > > > > > and
>> > > > > > > >     >> >>> > > try to
>> > > > > > > >     >> >>> > match
>> > > > > > > >     >> >>> > > the requester with their closest
geographical
>> > > > mirror.
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> > > Thoughts?
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> > > *Will STEVENS*
>> > > > > > > >     >> >>> > > Lead Developer
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep
>> > > Vittal <
>> > > > > > > >     >> >>> chiradeepv@gmail.com
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> > > wrote:
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> > > > What steps are needed to set up a mirror?
>> > > > > > > > What
>> > > > does
>> > > > > > > Infra
>> > > > > > > > need
>> > > > > > > >     >> to do?
>> > > > > > > >     >> >>> > Has
>> > > > > > > >     >> >>> > > > anybody filed a ticket with Infra?
>> > > > > > > >     >> >>> > > >
>> > > > > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja
>> > Pullela <
>> > > > > > > >     >> >>> > > > raja.pullela@accelerite.com>
>> > > > > > > >     >> >>> > > > wrote:
>> > > > > > > >     >> >>> > > >
>> > > > > > > >     >> >>> > > > > Hi will,
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > I believe, we didn’t get to close
>> > > > > > > > ‘getting a
>> > > > > mirror
>> > > > > > on
>> > > > > > > > Apache’
>> > > > > > > >     >> >>> > because
>> > > > > > > >     >> >>> > > we
>> > > > > > > >     >> >>> > > > > needed someone on the Apache Infra side
>> > > > > > > > to
>> > > close
>> > > > > > this.
>> > > > > > > > BTW,
>> > > > > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux
>> > > > > > > > manages
>> > > > this?)
>> > > > > > has
>> > > > > > > >     >> >>> > > > > all/most of
>> > > > > > > >     >> >>> > the
>> > > > > > > >     >> >>> > > > > content.  Once we can close on the
Apache
>> > > mirror
>> > > > > for
>> > > > > > > > hosting
>> > > > > > > >     >> >>> > > > > the
>> > > > > > > >     >> >>> > > > content, I
>> > > > > > > >     >> >>> > > > > can help assist getting the content
there.
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > For now, we have replicated the
>> > > > > download.cloud.com
>> > > > > > > > content to
>> > > > > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
>> > > > > > > >     >> >>> > > > > Also, we are working on a set of
>> > > steps/procedure
>> > > > > to
>> > > > > > > > help with
>> > > > > > > >     >> >>> > > > > this change.  I will update everyone in
>> > > > > > > > about
>> > > a
>> > > > > > week’s
>> > > > > > > > time
>> > > > > > > >     >> >>> > > > > on the
>> > > > > > > >     >> >>> > details.
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > Best,
>> > > > > > > >     >> >>> > > > > Raja Pullela
>> > > > > > > >     >> >>> > > > > Engineering Team,
>> > > > > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa
>> > > > > > > > Clara,
>> > > > CA,
>> > > > > > > 95054
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "
>> > > williamstevens@gmail.com
>> > > > > on
>> > > > > > > > behalf of
>> > > > > > > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com
>> > > > > > > > on
>> > > > behalf
>> > > > > > of
>> > > > > > > >     >> >>> > wstevens@cloudops.com>
>> > > > > > > >     >> >>> > > > > wrote:
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > unfortunately the template mirror
>> > > > > > > > conversation
>> > > > got
>> > > > > > > > caught up
>> > > > > > > >     >> >>> > > > > in
>> > > > > > > >     >> >>> > details
>> > > > > > > >     >> >>> > > > and
>> > > > > > > >     >> >>> > > > > nobody took the lead on implementing a
>> > > solution.
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > citrix has been pinging me every couple
>> > > > > > > > months
>> > > > to
>> > > > > > say
>> > > > > > > > 'dude,
>> > > > > > > >     >> >>> > > > > we
>> > > > > > > >     >> >>> need
>> > > > > > > >     >> >>> > to
>> > > > > > > >     >> >>> > > > > remove the dependency on
>> > > > > > > > download.citrix.com
>> > > ',
>> > > > > but
>> > > > > > i
>> > > > > > > > have not
>> > > > > > > >     >> >>> > > > > had
>> > > > > > > >     >> >>> > the
>> > > > > > > >     >> >>> > > > > cycles to get in and solve the problem.
>> > > > > > > > the
>> > > > > > shutdown
>> > > > > > > > of that
>> > > > > > > >     >> >>> > > > > is
>> > > > > > > >     >> >>> > > imminent
>> > > > > > > >     >> >>> > > > > right now, so we need to solve it asap.
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > *Will STEVENS*
>> > > > > > > >     >> >>> > > > > Lead Developer
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul
>> > Angus <
>> > > > > > > >     >> >>> > paul.angus@shapeblue.com
>> > > > > > > >     >> >>> > > >
>> > > > > > > >     >> >>> > > > > wrote:
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > > Hi Nathan,
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > Ideally, if you put the template
>> > > > > > > > location in
>> > > > (or
>> > > > > > use
>> > > > > > > a
>> > > > > > > >     >> >>> > > > > > template
>> > > > > > > >     >> >>> > > defined
>> > > > > > > >     >> >>> > > > > > in)  test_data.py then the actual
>> > > > > > > > location
>> > > can
>> > > > > be
>> > > > > > > >     >> >>> > > > > > overridden by
>> > > > > > > >     >> >>> > > anyone
>> > > > > > > >     >> >>> > > > > > testing.
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > For Trillian, we've copied all of the
>> > > > templates
>> > > > > > that
>> > > > > > > > people
>> > > > > > > >     >> >>> > > > > > have
>> > > > > > > >     >> >>> > > define
>> > > > > > > >     >> >>> > > > > to
>> > > > > > > >     >> >>> > > > > > a local repo and then replace the URLs
>> > > > > > > > in test_data.py to
>> > > > > > > >     >> >>> > > > > > reduce
>> > > > > > > >     >> >>> > > > > bandwidth
>> > > > > > > >     >> >>> > > > > > use and download times.
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > Ie:
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >             "bootableIso":
>> > > > > > > >     >> >>> > > > > >                 {
>> > > > > > > >     >> >>> > > > > >                     "displaytext":
"Test
>> > > > > Bootable
>> > > > > > > > ISO",
>> > > > > > > >     >> >>> > > > > >                     "name": "testISO",
>> > > > > > > >     >> >>> > > > > >                     "bootable": True,
>> > > > > > > >     >> >>> > > > > >                     "ispublic": False,
>> > > > > > > >     >> >>> > > > > >                     "url": "{{
>> > > > > > > marvin_images_location
>> > > > > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
>> > > > > > > >     >> >>> > > > > >                     "ostype": 'Other
>> Linux
>> > > > > > > (64-bit)',
>> > > > > > > >     >> >>> > > > > >                     "mode":
>> > 'HTTP_DOWNLOAD'
>> > > > > > > >     >> >>> > > > > >         },
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > I thought that we had come up with a
>> > > solution
>> > > > > for
>> > > > > > > >     >> >>> > download.cloud.com
>> > > > > > > >     >> >>> > > ,
>> > > > > > > >     >> >>> > > > by
>> > > > > > > >     >> >>> > > > > > having a mirrorlist hosted in
Community
>> > > Apache
>> > > > > > > > 'space' with
>> > > > > > > >     >> >>> anyone
>> > > > > > > >     >> >>> > > able
>> > > > > > > >     >> >>> > > > > to
>> > > > > > > >     >> >>> > > > > > out themselves forward as a mirror.
>> > > > > > > >     >> >>> > > > > > But I must admit I lost track of
>> > > > > > > > whether
>> > > > anyone
>> > > > > > made
>> > > > > > > > the
>> > > > > > > >     >> >>> requisite
>> > > > > > > >     >> >>> > > > > changes
>> > > > > > > >     >> >>> > > > > > in code....
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > Kind regards,
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > Paul Angus
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > paul.angus@shapeblue.com
>> > > > > > > >     >> >>> > > > > > www.shapeblue.com
>> > > > > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden,
London
>> > > WC2N
>> > > > > > 4HSUK
>> > > > > > > >     >> >>> > > > > > @shapeblue
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > -----Original Message-----
>> > > > > > > >     >> >>> > > > > > From: williamstevens@gmail.com
>> > > > > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
>> > > > > > > >     >> >>> > On
>> > > > > > > >     >> >>> > > > > > Behalf Of Will Stevens
>> > > > > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
>> > > > > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
>> > > > > > > >     >> >>> > > > > > Subject: Re: Modern template hosting
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > this is a hard questions.  in general,
>> > > > > > > > we
>> > > > should
>> > > > > > be
>> > > > > > > > setting
>> > > > > > > >     >> >>> > > > > > up a
>> > > > > > > >     >> >>> > > mirror
>> > > > > > > >     >> >>> > > > > on
>> > > > > > > >     >> >>> > > > > > some cloudstack/apache domain and then
>> > > mirror
>> > > > to
>> > > > > > > other
>> > > > > > > >     >> >>> > > > > > provided
>> > > > > > > >     >> >>> > > > > templates.
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > we MUST come up with a solution to
>> > deprecate
>> > > > > > > >     >> >>> > > > > > 'download.cloud.com
>> > > > > > > >     >> >>> ',
>> > > > > > > >     >> >>> > > > that
>> > > > > > > >     >> >>> > > > > > is going to be going away any day now.
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > i don't know the right way to solve
>> > > > > > > > this to
>> > > be
>> > > > > > > > honest, but
>> > > > > > > >     >> >>> > > > > > if you
>> > > > > > > >     >> >>> > > have
>> > > > > > > >     >> >>> > > > > > ideas, i am willing to help.
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > *Will STEVENS*
>> > > > > > > >     >> >>> > > > > > Lead Developer
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM,
>> > > > > > > > Nathan
>> > > > > Johnson <
>> > > > > > > >     >> >>> njohnson@ena.com
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> > > > > wrote:
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > > So not to re-open a can of worms,
but
>> > > > > > > > I’m
>> > > > in a
>> > > > > > > > situation
>> > > > > > > >     >> >>> > > > > > > where
>> > > > > > > >     >> >>> I
>> > > > > > > >     >> >>> > > need
>> > > > > > > >     >> >>> > > > > > > to come up with a Marvin component
>> > > > > > > > test
>> > > that
>> > > > > > > > depends on a
>> > > > > > > >     >> >>> > template
>> > > > > > > >     >> >>> > > > > > > based on a kernel that’s relatively
>> > > > > > > > new,
>> > > > i.e.,
>> > > > > > > > newer than
>> > > > > > > >     >> >>> Centos
>> > > > > > > >     >> >>> > > 5.3
>> > > > > > > >     >> >>> > > > /
>> > > > > > > >     >> >>> > > > > > Ubuntu 10.04 .
>> > > > > > > >     >> >>> > > > > > > I see openvm.eu has a suitable
>> > > > > > > > template
>> > > > > (Ubuntu
>> > > > > > > > 16.0.4
>> > > > > > > >     >> >>> > > > > > > for
>> > > > > > > >     >> >>> KVM),
>> > > > > > > >     >> >>> > > but
>> > > > > > > >     >> >>> > > > > > > from looking at the thread
"Migrating
>> > > > > CloudStack
>> > > > > > > > content
>> > > > > > > >     >> >>> > > > > > > from download.cloud.com” it looks
>> > > > > > > > like
>> > > > there
>> > > > > is
>> > > > > > > >     >> >>> > > > > > > resistance to using
>> > > > > > > >     >> >>> > > this
>> > > > > > > >     >> >>> > > > at
>> > > > > > > >     >> >>> > > > > > > least for hosting system vm
templates
>> > > > > > > > over
>> > > > > > > concerns
>> > > > > > > > of
>> > > > > > > >     >> >>> > neutrality.
>> > > > > > > >     >> >>> > > > > > > Would this be suitable for a
>> > > > > > > > component
>> > > test?
>> > > > > If
>> > > > > > > > not,
>> > > > > > > >     >> >>> > > > > > > what is a
>> > > > > > > >     >> >>> > > > > “blessed”
>> > > > > > > >     >> >>> > > > > > > template location?
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > > Thanks in advance!
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > > Nathan Johnson
>> > > > > > > >     >> >>> > > > > > > R&D Engineer
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12
>> > > > Nashville,
>> > > > > TN
>> > > > > > > > 37211
>> > > > > > > >     >> >>> > > > > > > General Office: 615-312-6000
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > > website | blog | support
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > DISCLAIMER
>> > > > > > > >     >> >>> > > > > ==========
>> > > > > > > >     >> >>> > > > > This e-mail may contain privileged and
>> > > > > confidential
>> > > > > > > >     >> >>> > > > > information
>> > > > > > > >     >> >>> which
>> > > > > > > >     >> >>> > > is
>> > > > > > > >     >> >>> > > > > the property of Accelerite, a Persistent
>> > > Systems
>> > > > > > > > business. It
>> > > > > > > >     >> >>> > > > > is
>> > > > > > > >     >> >>> > > intended
>> > > > > > > >     >> >>> > > > > only for the use of the individual or
>> > > > > > > > entity
>> > > to
>> > > > > > which
>> > > > > > > > it is
>> > > > > > > >     >> >>> > addressed.
>> > > > > > > >     >> >>> > > If
>> > > > > > > >     >> >>> > > > > you are not the intended recipient, you
>> > > > > > > > are
>> > > not
>> > > > > > > > authorized to
>> > > > > > > >     >> >>> > > > > read,
>> > > > > > > >     >> >>> > > > retain,
>> > > > > > > >     >> >>> > > > > copy, print, distribute or use this
>> message.
>> > > If
>> > > > > you
>> > > > > > > have
>> > > > > > > >     >> >>> > > > > received
>> > > > > > > >     >> >>> > this
>> > > > > > > >     >> >>> > > > > communication in error, please notify
the
>> > > sender
>> > > > > and
>> > > > > > > > delete
>> > > > > > > >     >> >>> > > > > all
>> > > > > > > >     >> >>> > copies
>> > > > > > > >     >> >>> > > of
>> > > > > > > >     >> >>> > > > > this message. Accelerite, a Persistent
>> > > > > > > > Systems
>> > > > > > > business
>> > > > > > > > does
>> > > > > > > >     >> >>> > > > > not
>> > > > > > > >     >> >>> > accept
>> > > > > > > >     >> >>> > > > any
>> > > > > > > >     >> >>> > > > > liability for virus infected mails.
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > >
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> >
>> > > > > > > >     >> >>>
>> > > > > > > >     >> >
>> > > > > > > >     >> >abhinandan.prateek@shapeblue.com
>> > > > > > > >     >> >www.shapeblue.com
>> > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > > > > @shapeblue
>> > > > > > > >     >> >
>> > > > > > > >     >> >
>> > > > > > > >     >> >
>> > > > > > > >     >>
>> > > > > > > >     >> abhinandan.prateek@shapeblue.com
>> > > > > > > >     >> www.shapeblue.com
>> > > > > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > > > > > > >     >> @shapeblue
>> > > > > > > >     >>
>> > > > > > > >     >>
>> > > > > > > >     >>
>> > > > > > > >     >>
>> > > > > > > >     >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > daan.hoogland@shapeblue.com
>> > > > > > > > www.shapeblue.com
>> > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > > > > > > > @shapeblue
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > Rafael Weingärtner
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Rafael Weingärtner
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>

Re: Modern template hosting

Posted by Erik Weber <te...@gmail.com>.

IMHO; implement it as simple as possible, use DNS RR and assume/expect
mirror hosts to use the same path.


Yes I know; DNS RR isn't bulletproof, but as already mentioned in this
thread - mirrors rarely go down and they aren't used all that often
(primarily seeding).
We can keep a low TTL on the record so that we're able to remove a
mirror that goes down for a significant period of time (more than
hours).

And yes I know, some operators might find it a burden having to set up
a new vhost with another document root, but tbh if they don't care
enough to do it we'll manage without that mirror.

-- 
Erik

On Sat, Mar 4, 2017 at 12:09 AM, Will Stevens <wi...@gmail.com> wrote:
> I agree with Paul. Look at the list of things they have to
> learn/master/care about. We don't want to add to that list, we want to
> remove from that list. Make the project/product more accessible. Reduce the
> barrier to entry...
>
> On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:
>
>> Seriously?
>> apt-get install apache2
>>
>> To install CloudStack, they need to know
>> - DB installation, perhaps some SQL
>> - VLAN configuration on their switches
>> - Ins-and-outs between port forwarding, static ips,
>> - NFS
>> - package management
>> - VHDs, qcow2. vmdk
>> - hyperviosrs
>> - and on and on.
>>
>> If they can figure all that out and not figure out a web server, they can
>> always come to the mailing list.
>>
>>
>>
>>
>> On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <pa...@shapeblue.com>
>> wrote:
>>
>> > The issue is not with supporting highly experienced cloud operators. It
>> is
>> > an issue for new users and other 'relatively' inexperienced operators.
>> >
>> > I have helped enough newbies and cloud operators who have been running
>> > their cloud for a while to know that the barriers to entry are too high
>> as
>> > they are. And telling anyone that they need to create a web server so
>> that
>> > they can add their initial template to get started or in order to create
>> a
>> > new zone, just isn't going to fly.
>> >
>> > I'm a huge advocate of the 'download.cloudstack.org' endpoint which the
>> > community can add/remove mirrors to or from for system VMs or built-in
>> > templates.  Can the same system be used for binary repos ? although there
>> > is an added complication of redist vs no-redist there...
>> >
>> >
>> >
>> >
>> >
>> >
>> > paul.angus@shapeblue.com
>> > www.shapeblue.com
>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > @shapeblue
>> >
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: Chiradeep Vittal [mailto:chiradeepv@gmail.com]
>> > Sent: 03 March 2017 18:28
>> > To: dev <de...@cloudstack.apache.org>
>> > Subject: Re: Modern template hosting
>> >
>> > I do feel like this is early optimization. Mirrors rarely fail. I'd
>> expect
>> > a single web server hosted on Apache Infra without any monitors to fail
>> > more often than a mirror. We already expect Wido's systemvm repository to
>> > be up all the time. And it has been. Similarly, I don't believe Nux's
>> > repository has ever been down. And if Accelerite wants to host on S3,
>> that
>> > one is pretty solid as well.
>> >
>> > This is an infrequent operation in a cloud. After the cloud is installed,
>> > the download servers are only needed for a new zone. If we trust the user
>> > to run a cloud, surely he/she can run a web server to serve some built-in
>> > templates. And if her cloud is successful, she needs to figure out how to
>> > host her templates anyway and not rely on 3rd parties.
>> >
>> >
>> > On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <ws...@cloudops.com>
>> > wrote:
>> >
>> > > 1) If the legacy implementations do not support redirects, that does
>> > > cause a problem.  A potential solution in that case is to have the web
>> > > server actually proxy the download, but that is not ideal and I would
>> > > like to avoid it if possible.  Thanks for bringing that up Chiradeep.
>> > >
>> > > 2) I think we need to have a single URL which people can target.  Once
>> > > they make the switch to the new URL, we want the implementation to be
>> > > able to handle mirror failures without affecting the end client.  We
>> > > want to avoid the situation where an ACS user will ever have to change
>> > > this URL more than once.  Mirror failures SHOULD NOT affect the ACS
>> > > users assuming there is still at least one mirror who can serve the
>> > requested resource.
>> > >
>> > > These are obviously my personal opinions and others will probably have
>> > > differing opinions.
>> > >
>> > > *Will STEVENS*
>> > > Lead Developer
>> > >
>> > > <https://goo.gl/NYZ8KK>
>> > >
>> > > On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal
>> > > <ch...@gmail.com>
>> > > wrote:
>> > >
>> > > > 1. If you are targeting legacy installations, they are not able to
>> > > > follow redirects. The line of code that added this capability was
>> > > > added on
>> > > 11/16.
>> > > > 2. If you trust the users to edit the database to change the URL,
>> > > > you can trust them to change it to anything. Just document a known
>> > > > list of good template locations.
>> > > >
>> > > > On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <ws...@cloudops.com>
>> > > > wrote:
>> > > >
>> > > > > So the main issue I see with this is the following.
>> > > > >
>> > > > > This implementation is designed to target legacy installations
>> > > > > which
>> > > will
>> > > > > be affected when download.cloud.com disappears.  These people will
>> > > need
>> > > > to
>> > > > > make a DB change to replace the 'download.cloud.com' with some
>> > > > > other
>> > > url
>> > > > > (maybe 'download.cloudstack.org').  Once the DB has been updated,
>> > > > > we
>> > > can
>> > > > > not expect anything else of the client, they need to be able to
>> > > > > just continue operation as they were without any need to rebuild
>> > > > > or upgrade their ACS.
>> > > > >
>> > > > > If we try to force the decision to the client, then we break the
>> > > > > legacy implementations.  Since the SSVM is likely going to be the
>> > > > > client in
>> > > some
>> > > > > cases, and since it does not already have the logic to handle the
>> > > > > 300 approach correctly, I feel like it is not a viable solution
>> > > > > for the
>> > > > legacy
>> > > > > deployments.
>> > > > >
>> > > > > In addition to that.  I don't think the client is in any better
>> > > position
>> > > > to
>> > > > > make the mirror decision than I am.  I am likely able to make a
>> > > > > more 'educated' decision than the client would be able to because
>> > > > > I can do different tests on the endpoint(s) before making a final
>> > decision.
>> > > > >
>> > > > > Does this make sense or am I missing something obvious?
>> > > > >
>> > > > > *Will STEVENS*
>> > > > > Lead Developer
>> > > > >
>> > > > > <https://goo.gl/NYZ8KK>
>> > > > >
>> > > > > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
>> > > > > rafaelweingartner@gmail.com> wrote:
>> > > > >
>> > > > > > I think I understood you, but I did not understand you.
>> > > > > >
>> > > > > > Let me see if I can create an example to illustrate how we could
>> > > > > > do
>> > > > using
>> > > > > > the HTTP 300 code.
>> > > > > >
>> > > > > > The HTTP 300 code indicates clients that he/she/it has multiple
>> > > choices
>> > > > > to
>> > > > > > access the requested resource. If the request is a HEAD, this is
>> > > > > > the
>> > > > only
>> > > > > > thing that the server returns. If the request if a GET, then the
>> > > server
>> > > > > > returns the choices for the client. The response type is defined
>> > > > > > on
>> > > the
>> > > > > > server based on the Content-Type, user agent and others.
>> > > > > >
>> > > > > > Let’s say the server receives a GET request and informs the
>> > > > content-type
>> > > > > as
>> > > > > > JSON (application/JSON). Your application could get the mirrors
>> > > > > > list,
>> > > > do
>> > > > > > the appending process for the requested resource path, and then
>> > > return
>> > > > a
>> > > > > > JSON array with the possible mirror the client can use. Then,
>> > > > > > the
>> > > > client
>> > > > > > gets this list and does the processing required to select a
>> mirror.
>> > > > > >
>> > > > > > Now let’s say that a human access the link using a browser.
>> > > > > > Then, the server should receive something like (text/plain) as
>> > > > > > the
>> > > content-type;
>> > > > I
>> > > > > > think depending on the browser this field may differ a little
>> > > > > > (it
>> > > would
>> > > > > > require some checking). Anyways, the server detects that it is a
>> > > > “human”
>> > > > > > requesting the resource, then we could serve an HTML page with a
>> > > > > Javascript
>> > > > > > that uses the mirror list. This Javascript could do some
>> > > > > > checking and choose the best mirror for that specific.
>> > > > > >
>> > > > > > Why do I think this approach is interesting?
>> > > > > > The application you developed would be used only to retrieve
>> > > > > > valid
>> > > and
>> > > > > > trusted sources of system VM images (a trusted repository for
>> > > > > > mirrors
>> > > > > that
>> > > > > > we as a community vouch for).  Also, we shift the decision
>> > > > > > process regarding mirrors from the server to the client. Then,
>> > > > > > it is up to
>> > > > > clients
>> > > > > > to select mirrors, and not up to us (or some of our
>> > implementations).
>> > > > > >
>> > > > > > Did this help?
>> > > > > > BTW: I am not saying I am against the way you proposed, which
>> > > > > > would
>> > > > work
>> > > > > > fine. It is merely a suggestion using a different perspective.
>> > > > > >
>> > > > > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
>> > > > williamstevens@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > @rafael: in general, I don't think the client should ever be
>> > > making a
>> > > > > > > choice about a mirror. We have to assume we are working with a
>> > > > scripted
>> > > > > > > application and anything hard coded in that implementation on
>> > > > > > > the
>> > > > > client
>> > > > > > > side is a risk.
>> > > > > > >
>> > > > > > > I may not be understanding the use of the 300 approach, so I
>> > > > > > > will
>> > > > > > research
>> > > > > > > it to see if I can make it fit.
>> > > > > > >
>> > > > > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
>> > > > > > rafaelweingartner@gmail.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > Will great job.
>> > > > > > >
>> > > > > > > I had the same doubt as Daan.
>> > > > > > >
>> > > > > > > Have you considered using HTTP 300 working mode? Then, we
>> > > > > > > could let
>> > > > the
>> > > > > > > client decide which mirror is the best(closest?). Section
>> > “10.3.1”
>> > > of
>> > > > > [1]
>> > > > > > > talks about it; this would put the pressure on deciding upon a
>> > > mirror
>> > > > > on
>> > > > > > > the client side, but I think it would be fair.
>> > > > > > >
>> > > > > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
>> > > > > > >
>> > > > > > > PS: You really like very short variables names!
>> > > > > > >
>> > > > > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
>> > > > > > daan.hoogland@shapeblue.com
>> > > > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Nice little thing Will,
>> > > > > > > >
>> > > > > > > > One question: if I read the code correctly it ‘go’es and
>> > > > > > > > tries
>> > > all
>> > > > > > > mirrors
>> > > > > > > > at once and whichever responses first is redirected to the
>> > > client.
>> > > > > This
>> > > > > > > > might well be the same every time. This might be the one
>> > > > > > > > closest
>> > > to
>> > > > > the
>> > > > > > > > mirror302 and not to the client. Is that correct or did I
>> > > > > > > > miss a
>> > > > > > weighing
>> > > > > > > > algorithm hidden in there?
>> > > > > > > >
>> > > > > > > > Good coding,
>> > > > > > > >
>> > > > > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
>> > > > > > > >
>> > > > > > > >     Hey All,
>> > > > > > > >     Please review this repo and tell me what you think:
>> > > > > > > >     https://github.com/swill/mirror302
>> > > > > > > >
>> > > > > > > >     Let me know if you have questions or you would like me
>> > > > > > > > to
>> > > make
>> > > > > > > changes
>> > > > > > > >     before I bring this topic up with ASF Infra and open a
>> > > > > > > > ticket
>> > > > to
>> > > > > > get
>> > > > > > > > this
>> > > > > > > >     implemented.
>> > > > > > > >
>> > > > > > > >     Cheers,
>> > > > > > > >
>> > > > > > > >     *Will Stevens*
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
>> > > > > > wstevens@cloudops.com
>> > > > > > > >
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > >     > I am building a short term solution right now.  I
>> > > > > > > > hopefully
>> > > > > will
>> > > > > > > > have the
>> > > > > > > >     > building blocks in place by the weekend so we can
>> > > > > > > > start
>> > > > working
>> > > > > > > with
>> > > > > > > > Infra
>> > > > > > > >     > to get it in place.
>> > > > > > > >     >
>> > > > > > > >     > We will have a web server which we can point something
>> > like
>> > > > > > > >     > downloads.cloudstack.org (or whatever url) at.  Then
>> > > > > > > > we
>> > > will
>> > > > > > have
>> > > > > > > > an ASF
>> > > > > > > >     > repo which tracks a mirror list and exposes a static
>> > > > > > > > site
>> > > to
>> > > > be
>> > > > > > > able
>> > > > > > > > to
>> > > > > > > >     > query the mirror list.
>> > > > > > > >     >
>> > > > > > > >     > The web server will receive a request and will do a
>> > > > > > > > 302
>> > > > > redirect
>> > > > > > to
>> > > > > > > > the
>> > > > > > > >     > appropriate resource on one of the mirrors.
>> > > > > > > >     >
>> > > > > > > >     > This gets us started.  Legacy environments will have
>> > > > > > > > to do
>> > > a
>> > > > DB
>> > > > > > > > change to
>> > > > > > > >     > change from 'download.cloud.com' to '
>> > > > downloads.cloudstack.org'
>> > > > > > (or
>> > > > > > > >     > whatever), but otherwise it should be pretty simple.
>> > > > > > > >     >
>> > > > > > > >     > We can improve how we deliver templates going forward,
>> > > > > > > > but
>> > > > this
>> > > > > > > > obviously
>> > > > > > > >     > requires some discussion still.  I will try to buy us
>> > > > > > > > some
>> > > > time
>> > > > > > > with
>> > > > > > > > an
>> > > > > > > >     > implementation which solves our problems today...
>> > > > > > > >     >
>> > > > > > > >     > *Will STEVENS*
>> > > > > > > >     > Lead Developer
>> > > > > > > >     >
>> > > > > > > >     > <https://goo.gl/NYZ8KK>
>> > > > > > > >     >
>> > > > > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
>> > > > > > > >     > abhinandan.prateek@shapeblue.com> wrote:
>> > > > > > > >     >
>> > > > > > > >     >>
>> > > > > > > >     >> Initial seeding is a manual step and that is not
>> > > > > > > > going to
>> > > > > change
>> > > > > > > in
>> > > > > > > > near
>> > > > > > > >     >> future. A handy list of official places from where
>> > > > > > > > these
>> > > > > > templates
>> > > > > > > > can be
>> > > > > > > >     >> downloaded won’t harm this part of the installation.
>> > > > > > > > Just
>> > > > > allow
>> > > > > > > > anyone
>> > > > > > > >     >> credible who follows democratic process to publish
>> > > > > > > > their
>> > > > > > templates
>> > > > > > > > with
>> > > > > > > >     >> apache’s blessings.
>> > > > > > > >     >>
>> > > > > > > >     >>
>> > > > > > > >     >> Coming to urls hard coded in db, the popular
>> > > > > > > > suggestion is
>> > > > to
>> > > > > > > point
>> > > > > > > > them
>> > > > > > > >     >> to mirrors. This looks good but this is going to take
>> > > > > > > > some
>> > > > > > effort
>> > > > > > > > that has
>> > > > > > > >     >> not materialised yet. Once this change is made it
>> > > > > > > > will be
>> > > > > going
>> > > > > > in
>> > > > > > > > some
>> > > > > > > >     >> future release like 4.11; the problem still remains
>> > > > > > > > for
>> > > the
>> > > > > > > > releases that
>> > > > > > > >     >> are already out there. In case access to
>> > > download.cloud.com
>> > > > > is
>> > > > > > > > dropped
>> > > > > > > >     >> at some point we will have to document the procedure
>> > > > > > > > such
>> > > > that
>> > > > > > > > people know
>> > > > > > > >     >> how to make these stable releases work. That is where
>> > > > > > > > the
>> > > > > > > procedure
>> > > > > > > >     >> documented here https://shankerbalan.net/blog/
>> > > > > > > >     >> seed-cloudstack-templates-offline/ should be
>> > > > > > > > officially
>> > > > > adapted
>> > > > > > > > with a
>> > > > > > > >     >> marketplace to pick up templates of choosing. Maybe
>> > > > > > > > we can
>> > > > > have
>> > > > > > a
>> > > > > > > > pre-setup
>> > > > > > > >     >> script to make choices instead of written wiki. Note
>> > > > > > > > that
>> > > > this
>> > > > > > > will
>> > > > > > > > be
>> > > > > > > >     >> anyway needed by the older releases.
>> > > > > > > >     >>
>> > > > > > > >     >>
>> > > > > > > >     >>
>> > > > > > > >     >>
>> > > > > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
>> > > > paul.angus@shapeblue.com>
>> > > > > > > > wrote:
>> > > > > > > >     >>
>> > > > > > > >     >> >-1
>> > > > > > > >     >> >
>> > > > > > > >     >> >We are trying to get MORE people to use CloudStack.
>> > > > > > > > And
>> > > > > > raising
>> > > > > > > > the
>> > > > > > > >     >> barrier to entry does nothing to help that.
>> > > > > > > >     >> >
>> > > > > > > >     >> >
>> > > > > > > >     >> >Kind regards,
>> > > > > > > >     >> >
>> > > > > > > >     >> >Paul Angus
>> > > > > > > >     >> >
>> > > > > > > >     >> >paul.angus@shapeblue.com
>> > > > > > > >     >> >www.shapeblue.com
>> > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > > > > > > >     >> >@shapeblue
>> > > > > > > >     >> >
>> > > > > > > >     >> >
>> > > > > > > >     >> >
>> > > > > > > >     >> >
>> > > > > > > >     >> >-----Original Message-----
>> > > > > > > >     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@
>> > > > > > > shapeblue.com]
>> > > > > > > >     >> >Sent: 01 March 2017 09:39
>> > > > > > > >     >> >To: dev@cloudstack.apache.org
>> > > > > > > >     >> >Subject: Re: Modern template hosting
>> > > > > > > >     >> >
>> > > > > > > >     >> >+1
>> > > > > > > >     >> >
>> > > > > > > >     >> >After seeing this discussion running in circles
>> > > > > > > > several
>> > > > > times,
>> > > > > > I
>> > > > > > > > think
>> > > > > > > >     >> we should at least get started with the simplest
>> option.
>> > > > > > > >     >> >
>> > > > > > > >     >> >
>> > > > > > > >     >> >
>> > > > > > > >     >> >
>> > > > > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
>> > > > > > chiradeepv@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >     >> >
>> > > > > > > >     >> >>The alternative foolproof way is to simply not
>> > > > > > > > provide
>> > > the
>> > > > > > > > automatic
>> > > > > > > >     >> >>download.
>> > > > > > > >     >> >>
>> > > > > > > >     >> >>Just document the mirror list URL where the
>> > > > > > > > templates
>> > > can
>> > > > be
>> > > > > > > > found. I
>> > > > > > > >     >> >>don't know why people reject this option.
>> > > > > > > >     >> >>
>> > > > > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
>> > > > > > > templates-offline/
>> > > > > > > >     >> >>
>> > > > > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
>> > > > > > > >     >> >><wi...@gmail.com>
>> > > > > > > >     >> >>wrote:
>> > > > > > > >     >> >>
>> > > > > > > >     >> >>> OK. Thanks for the heads up.
>> > > > > > > >     >> >>>
>> > > > > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
>> > > > > > > > chiradeepv@gmail.com>
>> > > > > > > >     >> wrote:
>> > > > > > > >     >> >>>
>> > > > > > > >     >> >>> > Sounds workable. The downloader code in the
>> > > > > > > > SSVM
>> > > won't
>> > > > > > > follow
>> > > > > > > >     >> >>> > redirects I think.
>> > > > > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
>> > > > > > > 5511065fc20787619d
>> > > > > > > > 9cd0444
>> > > > > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/
>> > > storage/template/
>> > > > > > > >     >> >>> > HttpTemplateDownloader.java#L93
>> > > > > > > >     >> >>> > https://goo.gl/dSi0r5
>> > > > > > > >     >> >>> >  Might need to add
>> > > > > > > >     >> >>> > client.setRedirectStrategy(new
>> > > LaxRedirectStrategy());
>> > > > > > > >     >> >>> >
>> > > > > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
>> > > > > > > >     >> >>> > <ws...@cloudops.com>
>> > > > > > > >     >> >>> > wrote:
>> > > > > > > >     >> >>> >
>> > > > > > > >     >> >>> > > We haven't opened a ticket yet because we
>> > > > > > > > don't
>> > > > have a
>> > > > > > > > strategy
>> > > > > > > >     >> yet.
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> > > What do you guys think of this:
>> > > > > > > >     >> >>> > > - We setup a new github repo in the 'apache'
>> > > > > > > > org
>> > > > which
>> > > > > > > > consists
>> > > > > > > >     >> >>> > > of a
>> > > > > > > >     >> >>> > single
>> > > > > > > >     >> >>> > > file with a list of active/supported mirrors.
>> > > > > > > >     >> >>> > > - I write a small web server, distributed as
>> > > > > > > > a
>> > > > binary,
>> > > > > > > > which can
>> > > > > > > >     >> >>> > > be
>> > > > > > > >     >> >>> > hosted
>> > > > > > > >     >> >>> > > by ASF Infra.  This web server will query the
>> > > > current
>> > > > > > list
>> > > > > > > > of
>> > > > > > > >     >> >>> > > mirrors
>> > > > > > > >     >> >>> and
>> > > > > > > >     >> >>> > > will select one and then do a 302 redirect to
>> > > > > > > > that
>> > > > > > mirror.
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> > > The act of 'choosing' a mirror could be done
>> > > > > > > > in a
>> > > > > number
>> > > > > > > of
>> > > > > > > > ways.
>> > > > > > > >     >> >>> > > - If we want to define an order, then it
>> > > > > > > > could
>> > > just
>> > > > > try
>> > > > > > > > from the
>> > > > > > > >     >> >>> > > top of
>> > > > > > > >     >> >>> > the
>> > > > > > > >     >> >>> > > list and work its way down.  It would curl
>> > > > > > > > the
>> > > > target
>> > > > > to
>> > > > > > > > make
>> > > > > > > >     >> >>> > > sure it
>> > > > > > > >     >> >>> > gets
>> > > > > > > >     >> >>> > > a 200 and if it does, it would do a 302
>> > redirect.
>> > > > > > > >     >> >>> > > - Or, if we want to distribute the load
>> > > > > > > > across the
>> > > > > > > mirrors,
>> > > > > > > > we
>> > > > > > > >     >> >>> > > could
>> > > > > > > >     >> >>> pick
>> > > > > > > >     >> >>> > > from the list randomly.  Again, doing a curl
>> > > > > > > > to
>> > > > verify
>> > > > > > the
>> > > > > > > > mirror
>> > > > > > > >     >> >>> > > is up
>> > > > > > > >     >> >>> > and
>> > > > > > > >     >> >>> > > then doing a redirect.
>> > > > > > > >     >> >>> > > - If we want to get fancy, we could do a
>> > > > > > > > reverse
>> > > IP
>> > > > > > lookup
>> > > > > > > > and
>> > > > > > > >     >> >>> > > try to
>> > > > > > > >     >> >>> > match
>> > > > > > > >     >> >>> > > the requester with their closest geographical
>> > > > mirror.
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> > > Thoughts?
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> > > *Will STEVENS*
>> > > > > > > >     >> >>> > > Lead Developer
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep
>> > > Vittal <
>> > > > > > > >     >> >>> chiradeepv@gmail.com
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> > > wrote:
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> > > > What steps are needed to set up a mirror?
>> > > > > > > > What
>> > > > does
>> > > > > > > Infra
>> > > > > > > > need
>> > > > > > > >     >> to do?
>> > > > > > > >     >> >>> > Has
>> > > > > > > >     >> >>> > > > anybody filed a ticket with Infra?
>> > > > > > > >     >> >>> > > >
>> > > > > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja
>> > Pullela <
>> > > > > > > >     >> >>> > > > raja.pullela@accelerite.com>
>> > > > > > > >     >> >>> > > > wrote:
>> > > > > > > >     >> >>> > > >
>> > > > > > > >     >> >>> > > > > Hi will,
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > I believe, we didn’t get to close
>> > > > > > > > ‘getting a
>> > > > > mirror
>> > > > > > on
>> > > > > > > > Apache’
>> > > > > > > >     >> >>> > because
>> > > > > > > >     >> >>> > > we
>> > > > > > > >     >> >>> > > > > needed someone on the Apache Infra side
>> > > > > > > > to
>> > > close
>> > > > > > this.
>> > > > > > > > BTW,
>> > > > > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux
>> > > > > > > > manages
>> > > > this?)
>> > > > > > has
>> > > > > > > >     >> >>> > > > > all/most of
>> > > > > > > >     >> >>> > the
>> > > > > > > >     >> >>> > > > > content.  Once we can close on the Apache
>> > > mirror
>> > > > > for
>> > > > > > > > hosting
>> > > > > > > >     >> >>> > > > > the
>> > > > > > > >     >> >>> > > > content, I
>> > > > > > > >     >> >>> > > > > can help assist getting the content there.
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > For now, we have replicated the
>> > > > > download.cloud.com
>> > > > > > > > content to
>> > > > > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
>> > > > > > > >     >> >>> > > > > Also, we are working on a set of
>> > > steps/procedure
>> > > > > to
>> > > > > > > > help with
>> > > > > > > >     >> >>> > > > > this change.  I will update everyone in
>> > > > > > > > about
>> > > a
>> > > > > > week’s
>> > > > > > > > time
>> > > > > > > >     >> >>> > > > > on the
>> > > > > > > >     >> >>> > details.
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > Best,
>> > > > > > > >     >> >>> > > > > Raja Pullela
>> > > > > > > >     >> >>> > > > > Engineering Team,
>> > > > > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa
>> > > > > > > > Clara,
>> > > > CA,
>> > > > > > > 95054
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "
>> > > williamstevens@gmail.com
>> > > > > on
>> > > > > > > > behalf of
>> > > > > > > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com
>> > > > > > > > on
>> > > > behalf
>> > > > > > of
>> > > > > > > >     >> >>> > wstevens@cloudops.com>
>> > > > > > > >     >> >>> > > > > wrote:
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > unfortunately the template mirror
>> > > > > > > > conversation
>> > > > got
>> > > > > > > > caught up
>> > > > > > > >     >> >>> > > > > in
>> > > > > > > >     >> >>> > details
>> > > > > > > >     >> >>> > > > and
>> > > > > > > >     >> >>> > > > > nobody took the lead on implementing a
>> > > solution.
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > citrix has been pinging me every couple
>> > > > > > > > months
>> > > > to
>> > > > > > say
>> > > > > > > > 'dude,
>> > > > > > > >     >> >>> > > > > we
>> > > > > > > >     >> >>> need
>> > > > > > > >     >> >>> > to
>> > > > > > > >     >> >>> > > > > remove the dependency on
>> > > > > > > > download.citrix.com
>> > > ',
>> > > > > but
>> > > > > > i
>> > > > > > > > have not
>> > > > > > > >     >> >>> > > > > had
>> > > > > > > >     >> >>> > the
>> > > > > > > >     >> >>> > > > > cycles to get in and solve the problem.
>> > > > > > > > the
>> > > > > > shutdown
>> > > > > > > > of that
>> > > > > > > >     >> >>> > > > > is
>> > > > > > > >     >> >>> > > imminent
>> > > > > > > >     >> >>> > > > > right now, so we need to solve it asap.
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > *Will STEVENS*
>> > > > > > > >     >> >>> > > > > Lead Developer
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul
>> > Angus <
>> > > > > > > >     >> >>> > paul.angus@shapeblue.com
>> > > > > > > >     >> >>> > > >
>> > > > > > > >     >> >>> > > > > wrote:
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > > Hi Nathan,
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > Ideally, if you put the template
>> > > > > > > > location in
>> > > > (or
>> > > > > > use
>> > > > > > > a
>> > > > > > > >     >> >>> > > > > > template
>> > > > > > > >     >> >>> > > defined
>> > > > > > > >     >> >>> > > > > > in)  test_data.py then the actual
>> > > > > > > > location
>> > > can
>> > > > > be
>> > > > > > > >     >> >>> > > > > > overridden by
>> > > > > > > >     >> >>> > > anyone
>> > > > > > > >     >> >>> > > > > > testing.
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > For Trillian, we've copied all of the
>> > > > templates
>> > > > > > that
>> > > > > > > > people
>> > > > > > > >     >> >>> > > > > > have
>> > > > > > > >     >> >>> > > define
>> > > > > > > >     >> >>> > > > > to
>> > > > > > > >     >> >>> > > > > > a local repo and then replace the URLs
>> > > > > > > > in test_data.py to
>> > > > > > > >     >> >>> > > > > > reduce
>> > > > > > > >     >> >>> > > > > bandwidth
>> > > > > > > >     >> >>> > > > > > use and download times.
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > Ie:
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >             "bootableIso":
>> > > > > > > >     >> >>> > > > > >                 {
>> > > > > > > >     >> >>> > > > > >                     "displaytext": "Test
>> > > > > Bootable
>> > > > > > > > ISO",
>> > > > > > > >     >> >>> > > > > >                     "name": "testISO",
>> > > > > > > >     >> >>> > > > > >                     "bootable": True,
>> > > > > > > >     >> >>> > > > > >                     "ispublic": False,
>> > > > > > > >     >> >>> > > > > >                     "url": "{{
>> > > > > > > marvin_images_location
>> > > > > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
>> > > > > > > >     >> >>> > > > > >                     "ostype": 'Other
>> Linux
>> > > > > > > (64-bit)',
>> > > > > > > >     >> >>> > > > > >                     "mode":
>> > 'HTTP_DOWNLOAD'
>> > > > > > > >     >> >>> > > > > >         },
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > I thought that we had come up with a
>> > > solution
>> > > > > for
>> > > > > > > >     >> >>> > download.cloud.com
>> > > > > > > >     >> >>> > > ,
>> > > > > > > >     >> >>> > > > by
>> > > > > > > >     >> >>> > > > > > having a mirrorlist hosted in Community
>> > > Apache
>> > > > > > > > 'space' with
>> > > > > > > >     >> >>> anyone
>> > > > > > > >     >> >>> > > able
>> > > > > > > >     >> >>> > > > > to
>> > > > > > > >     >> >>> > > > > > out themselves forward as a mirror.
>> > > > > > > >     >> >>> > > > > > But I must admit I lost track of
>> > > > > > > > whether
>> > > > anyone
>> > > > > > made
>> > > > > > > > the
>> > > > > > > >     >> >>> requisite
>> > > > > > > >     >> >>> > > > > changes
>> > > > > > > >     >> >>> > > > > > in code....
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > Kind regards,
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > Paul Angus
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > paul.angus@shapeblue.com
>> > > > > > > >     >> >>> > > > > > www.shapeblue.com
>> > > > > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden, London
>> > > WC2N
>> > > > > > 4HSUK
>> > > > > > > >     >> >>> > > > > > @shapeblue
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > -----Original Message-----
>> > > > > > > >     >> >>> > > > > > From: williamstevens@gmail.com
>> > > > > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
>> > > > > > > >     >> >>> > On
>> > > > > > > >     >> >>> > > > > > Behalf Of Will Stevens
>> > > > > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
>> > > > > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
>> > > > > > > >     >> >>> > > > > > Subject: Re: Modern template hosting
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > this is a hard questions.  in general,
>> > > > > > > > we
>> > > > should
>> > > > > > be
>> > > > > > > > setting
>> > > > > > > >     >> >>> > > > > > up a
>> > > > > > > >     >> >>> > > mirror
>> > > > > > > >     >> >>> > > > > on
>> > > > > > > >     >> >>> > > > > > some cloudstack/apache domain and then
>> > > mirror
>> > > > to
>> > > > > > > other
>> > > > > > > >     >> >>> > > > > > provided
>> > > > > > > >     >> >>> > > > > templates.
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > we MUST come up with a solution to
>> > deprecate
>> > > > > > > >     >> >>> > > > > > 'download.cloud.com
>> > > > > > > >     >> >>> ',
>> > > > > > > >     >> >>> > > > that
>> > > > > > > >     >> >>> > > > > > is going to be going away any day now.
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > i don't know the right way to solve
>> > > > > > > > this to
>> > > be
>> > > > > > > > honest, but
>> > > > > > > >     >> >>> > > > > > if you
>> > > > > > > >     >> >>> > > have
>> > > > > > > >     >> >>> > > > > > ideas, i am willing to help.
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > *Will STEVENS*
>> > > > > > > >     >> >>> > > > > > Lead Developer
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM,
>> > > > > > > > Nathan
>> > > > > Johnson <
>> > > > > > > >     >> >>> njohnson@ena.com
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> > > > > wrote:
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > > > > So not to re-open a can of worms, but
>> > > > > > > > I’m
>> > > > in a
>> > > > > > > > situation
>> > > > > > > >     >> >>> > > > > > > where
>> > > > > > > >     >> >>> I
>> > > > > > > >     >> >>> > > need
>> > > > > > > >     >> >>> > > > > > > to come up with a Marvin component
>> > > > > > > > test
>> > > that
>> > > > > > > > depends on a
>> > > > > > > >     >> >>> > template
>> > > > > > > >     >> >>> > > > > > > based on a kernel that’s relatively
>> > > > > > > > new,
>> > > > i.e.,
>> > > > > > > > newer than
>> > > > > > > >     >> >>> Centos
>> > > > > > > >     >> >>> > > 5.3
>> > > > > > > >     >> >>> > > > /
>> > > > > > > >     >> >>> > > > > > Ubuntu 10.04 .
>> > > > > > > >     >> >>> > > > > > > I see openvm.eu has a suitable
>> > > > > > > > template
>> > > > > (Ubuntu
>> > > > > > > > 16.0.4
>> > > > > > > >     >> >>> > > > > > > for
>> > > > > > > >     >> >>> KVM),
>> > > > > > > >     >> >>> > > but
>> > > > > > > >     >> >>> > > > > > > from looking at the thread "Migrating
>> > > > > CloudStack
>> > > > > > > > content
>> > > > > > > >     >> >>> > > > > > > from download.cloud.com” it looks
>> > > > > > > > like
>> > > > there
>> > > > > is
>> > > > > > > >     >> >>> > > > > > > resistance to using
>> > > > > > > >     >> >>> > > this
>> > > > > > > >     >> >>> > > > at
>> > > > > > > >     >> >>> > > > > > > least for hosting system vm templates
>> > > > > > > > over
>> > > > > > > concerns
>> > > > > > > > of
>> > > > > > > >     >> >>> > neutrality.
>> > > > > > > >     >> >>> > > > > > > Would this be suitable for a
>> > > > > > > > component
>> > > test?
>> > > > > If
>> > > > > > > > not,
>> > > > > > > >     >> >>> > > > > > > what is a
>> > > > > > > >     >> >>> > > > > “blessed”
>> > > > > > > >     >> >>> > > > > > > template location?
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > > Thanks in advance!
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > > Nathan Johnson
>> > > > > > > >     >> >>> > > > > > > R&D Engineer
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12
>> > > > Nashville,
>> > > > > TN
>> > > > > > > > 37211
>> > > > > > > >     >> >>> > > > > > > General Office: 615-312-6000
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > > website | blog | support
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > > >
>> > > > > > > >     >> >>> > > > > >
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > > > DISCLAIMER
>> > > > > > > >     >> >>> > > > > ==========
>> > > > > > > >     >> >>> > > > > This e-mail may contain privileged and
>> > > > > confidential
>> > > > > > > >     >> >>> > > > > information
>> > > > > > > >     >> >>> which
>> > > > > > > >     >> >>> > > is
>> > > > > > > >     >> >>> > > > > the property of Accelerite, a Persistent
>> > > Systems
>> > > > > > > > business. It
>> > > > > > > >     >> >>> > > > > is
>> > > > > > > >     >> >>> > > intended
>> > > > > > > >     >> >>> > > > > only for the use of the individual or
>> > > > > > > > entity
>> > > to
>> > > > > > which
>> > > > > > > > it is
>> > > > > > > >     >> >>> > addressed.
>> > > > > > > >     >> >>> > > If
>> > > > > > > >     >> >>> > > > > you are not the intended recipient, you
>> > > > > > > > are
>> > > not
>> > > > > > > > authorized to
>> > > > > > > >     >> >>> > > > > read,
>> > > > > > > >     >> >>> > > > retain,
>> > > > > > > >     >> >>> > > > > copy, print, distribute or use this
>> message.
>> > > If
>> > > > > you
>> > > > > > > have
>> > > > > > > >     >> >>> > > > > received
>> > > > > > > >     >> >>> > this
>> > > > > > > >     >> >>> > > > > communication in error, please notify the
>> > > sender
>> > > > > and
>> > > > > > > > delete
>> > > > > > > >     >> >>> > > > > all
>> > > > > > > >     >> >>> > copies
>> > > > > > > >     >> >>> > > of
>> > > > > > > >     >> >>> > > > > this message. Accelerite, a Persistent
>> > > > > > > > Systems
>> > > > > > > business
>> > > > > > > > does
>> > > > > > > >     >> >>> > > > > not
>> > > > > > > >     >> >>> > accept
>> > > > > > > >     >> >>> > > > any
>> > > > > > > >     >> >>> > > > > liability for virus infected mails.
>> > > > > > > >     >> >>> > > > >
>> > > > > > > >     >> >>> > > >
>> > > > > > > >     >> >>> > >
>> > > > > > > >     >> >>> >
>> > > > > > > >     >> >>>
>> > > > > > > >     >> >
>> > > > > > > >     >> >abhinandan.prateek@shapeblue.com
>> > > > > > > >     >> >www.shapeblue.com
>> > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > > > > @shapeblue
>> > > > > > > >     >> >
>> > > > > > > >     >> >
>> > > > > > > >     >> >
>> > > > > > > >     >>
>> > > > > > > >     >> abhinandan.prateek@shapeblue.com
>> > > > > > > >     >> www.shapeblue.com
>> > > > > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > > > > > > >     >> @shapeblue
>> > > > > > > >     >>
>> > > > > > > >     >>
>> > > > > > > >     >>
>> > > > > > > >     >>
>> > > > > > > >     >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > daan.hoogland@shapeblue.com
>> > > > > > > > www.shapeblue.com
>> > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > > > > > > > @shapeblue
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > Rafael Weingärtner
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Rafael Weingärtner
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>

Re: Modern template hosting

Posted by Will Stevens <wi...@gmail.com>.

I agree with Paul. Look at the list of things they have to
learn/master/care about. We don't want to add to that list, we want to
remove from that list. Make the project/product more accessible. Reduce the
barrier to entry...

On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:

> Seriously?
> apt-get install apache2
>
> To install CloudStack, they need to know
> - DB installation, perhaps some SQL
> - VLAN configuration on their switches
> - Ins-and-outs between port forwarding, static ips,
> - NFS
> - package management
> - VHDs, qcow2. vmdk
> - hyperviosrs
> - and on and on.
>
> If they can figure all that out and not figure out a web server, they can
> always come to the mailing list.
>
>
>
>
> On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <pa...@shapeblue.com>
> wrote:
>
> > The issue is not with supporting highly experienced cloud operators. It
> is
> > an issue for new users and other 'relatively' inexperienced operators.
> >
> > I have helped enough newbies and cloud operators who have been running
> > their cloud for a while to know that the barriers to entry are too high
> as
> > they are. And telling anyone that they need to create a web server so
> that
> > they can add their initial template to get started or in order to create
> a
> > new zone, just isn't going to fly.
> >
> > I'm a huge advocate of the 'download.cloudstack.org' endpoint which the
> > community can add/remove mirrors to or from for system VMs or built-in
> > templates.  Can the same system be used for binary repos ? although there
> > is an added complication of redist vs no-redist there...
> >
> >
> >
> >
> >
> >
> > paul.angus@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> > -----Original Message-----
> > From: Chiradeep Vittal [mailto:chiradeepv@gmail.com]
> > Sent: 03 March 2017 18:28
> > To: dev <de...@cloudstack.apache.org>
> > Subject: Re: Modern template hosting
> >
> > I do feel like this is early optimization. Mirrors rarely fail. I'd
> expect
> > a single web server hosted on Apache Infra without any monitors to fail
> > more often than a mirror. We already expect Wido's systemvm repository to
> > be up all the time. And it has been. Similarly, I don't believe Nux's
> > repository has ever been down. And if Accelerite wants to host on S3,
> that
> > one is pretty solid as well.
> >
> > This is an infrequent operation in a cloud. After the cloud is installed,
> > the download servers are only needed for a new zone. If we trust the user
> > to run a cloud, surely he/she can run a web server to serve some built-in
> > templates. And if her cloud is successful, she needs to figure out how to
> > host her templates anyway and not rely on 3rd parties.
> >
> >
> > On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <ws...@cloudops.com>
> > wrote:
> >
> > > 1) If the legacy implementations do not support redirects, that does
> > > cause a problem.  A potential solution in that case is to have the web
> > > server actually proxy the download, but that is not ideal and I would
> > > like to avoid it if possible.  Thanks for bringing that up Chiradeep.
> > >
> > > 2) I think we need to have a single URL which people can target.  Once
> > > they make the switch to the new URL, we want the implementation to be
> > > able to handle mirror failures without affecting the end client.  We
> > > want to avoid the situation where an ACS user will ever have to change
> > > this URL more than once.  Mirror failures SHOULD NOT affect the ACS
> > > users assuming there is still at least one mirror who can serve the
> > requested resource.
> > >
> > > These are obviously my personal opinions and others will probably have
> > > differing opinions.
> > >
> > > *Will STEVENS*
> > > Lead Developer
> > >
> > > <https://goo.gl/NYZ8KK>
> > >
> > > On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal
> > > <ch...@gmail.com>
> > > wrote:
> > >
> > > > 1. If you are targeting legacy installations, they are not able to
> > > > follow redirects. The line of code that added this capability was
> > > > added on
> > > 11/16.
> > > > 2. If you trust the users to edit the database to change the URL,
> > > > you can trust them to change it to anything. Just document a known
> > > > list of good template locations.
> > > >
> > > > On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <ws...@cloudops.com>
> > > > wrote:
> > > >
> > > > > So the main issue I see with this is the following.
> > > > >
> > > > > This implementation is designed to target legacy installations
> > > > > which
> > > will
> > > > > be affected when download.cloud.com disappears.  These people will
> > > need
> > > > to
> > > > > make a DB change to replace the 'download.cloud.com' with some
> > > > > other
> > > url
> > > > > (maybe 'download.cloudstack.org').  Once the DB has been updated,
> > > > > we
> > > can
> > > > > not expect anything else of the client, they need to be able to
> > > > > just continue operation as they were without any need to rebuild
> > > > > or upgrade their ACS.
> > > > >
> > > > > If we try to force the decision to the client, then we break the
> > > > > legacy implementations.  Since the SSVM is likely going to be the
> > > > > client in
> > > some
> > > > > cases, and since it does not already have the logic to handle the
> > > > > 300 approach correctly, I feel like it is not a viable solution
> > > > > for the
> > > > legacy
> > > > > deployments.
> > > > >
> > > > > In addition to that.  I don't think the client is in any better
> > > position
> > > > to
> > > > > make the mirror decision than I am.  I am likely able to make a
> > > > > more 'educated' decision than the client would be able to because
> > > > > I can do different tests on the endpoint(s) before making a final
> > decision.
> > > > >
> > > > > Does this make sense or am I missing something obvious?
> > > > >
> > > > > *Will STEVENS*
> > > > > Lead Developer
> > > > >
> > > > > <https://goo.gl/NYZ8KK>
> > > > >
> > > > > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
> > > > > rafaelweingartner@gmail.com> wrote:
> > > > >
> > > > > > I think I understood you, but I did not understand you.
> > > > > >
> > > > > > Let me see if I can create an example to illustrate how we could
> > > > > > do
> > > > using
> > > > > > the HTTP 300 code.
> > > > > >
> > > > > > The HTTP 300 code indicates clients that he/she/it has multiple
> > > choices
> > > > > to
> > > > > > access the requested resource. If the request is a HEAD, this is
> > > > > > the
> > > > only
> > > > > > thing that the server returns. If the request if a GET, then the
> > > server
> > > > > > returns the choices for the client. The response type is defined
> > > > > > on
> > > the
> > > > > > server based on the Content-Type, user agent and others.
> > > > > >
> > > > > > Let’s say the server receives a GET request and informs the
> > > > content-type
> > > > > as
> > > > > > JSON (application/JSON). Your application could get the mirrors
> > > > > > list,
> > > > do
> > > > > > the appending process for the requested resource path, and then
> > > return
> > > > a
> > > > > > JSON array with the possible mirror the client can use. Then,
> > > > > > the
> > > > client
> > > > > > gets this list and does the processing required to select a
> mirror.
> > > > > >
> > > > > > Now let’s say that a human access the link using a browser.
> > > > > > Then, the server should receive something like (text/plain) as
> > > > > > the
> > > content-type;
> > > > I
> > > > > > think depending on the browser this field may differ a little
> > > > > > (it
> > > would
> > > > > > require some checking). Anyways, the server detects that it is a
> > > > “human”
> > > > > > requesting the resource, then we could serve an HTML page with a
> > > > > Javascript
> > > > > > that uses the mirror list. This Javascript could do some
> > > > > > checking and choose the best mirror for that specific.
> > > > > >
> > > > > > Why do I think this approach is interesting?
> > > > > > The application you developed would be used only to retrieve
> > > > > > valid
> > > and
> > > > > > trusted sources of system VM images (a trusted repository for
> > > > > > mirrors
> > > > > that
> > > > > > we as a community vouch for).  Also, we shift the decision
> > > > > > process regarding mirrors from the server to the client. Then,
> > > > > > it is up to
> > > > > clients
> > > > > > to select mirrors, and not up to us (or some of our
> > implementations).
> > > > > >
> > > > > > Did this help?
> > > > > > BTW: I am not saying I am against the way you proposed, which
> > > > > > would
> > > > work
> > > > > > fine. It is merely a suggestion using a different perspective.
> > > > > >
> > > > > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
> > > > williamstevens@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > @rafael: in general, I don't think the client should ever be
> > > making a
> > > > > > > choice about a mirror. We have to assume we are working with a
> > > > scripted
> > > > > > > application and anything hard coded in that implementation on
> > > > > > > the
> > > > > client
> > > > > > > side is a risk.
> > > > > > >
> > > > > > > I may not be understanding the use of the 300 approach, so I
> > > > > > > will
> > > > > > research
> > > > > > > it to see if I can make it fit.
> > > > > > >
> > > > > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
> > > > > > rafaelweingartner@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > Will great job.
> > > > > > >
> > > > > > > I had the same doubt as Daan.
> > > > > > >
> > > > > > > Have you considered using HTTP 300 working mode? Then, we
> > > > > > > could let
> > > > the
> > > > > > > client decide which mirror is the best(closest?). Section
> > “10.3.1”
> > > of
> > > > > [1]
> > > > > > > talks about it; this would put the pressure on deciding upon a
> > > mirror
> > > > > on
> > > > > > > the client side, but I think it would be fair.
> > > > > > >
> > > > > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
> > > > > > >
> > > > > > > PS: You really like very short variables names!
> > > > > > >
> > > > > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
> > > > > > daan.hoogland@shapeblue.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Nice little thing Will,
> > > > > > > >
> > > > > > > > One question: if I read the code correctly it ‘go’es and
> > > > > > > > tries
> > > all
> > > > > > > mirrors
> > > > > > > > at once and whichever responses first is redirected to the
> > > client.
> > > > > This
> > > > > > > > might well be the same every time. This might be the one
> > > > > > > > closest
> > > to
> > > > > the
> > > > > > > > mirror302 and not to the client. Is that correct or did I
> > > > > > > > miss a
> > > > > > weighing
> > > > > > > > algorithm hidden in there?
> > > > > > > >
> > > > > > > > Good coding,
> > > > > > > >
> > > > > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
> > > > > > > >
> > > > > > > >     Hey All,
> > > > > > > >     Please review this repo and tell me what you think:
> > > > > > > >     https://github.com/swill/mirror302
> > > > > > > >
> > > > > > > >     Let me know if you have questions or you would like me
> > > > > > > > to
> > > make
> > > > > > > changes
> > > > > > > >     before I bring this topic up with ASF Infra and open a
> > > > > > > > ticket
> > > > to
> > > > > > get
> > > > > > > > this
> > > > > > > >     implemented.
> > > > > > > >
> > > > > > > >     Cheers,
> > > > > > > >
> > > > > > > >     *Will Stevens*
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> > > > > > wstevens@cloudops.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >     > I am building a short term solution right now.  I
> > > > > > > > hopefully
> > > > > will
> > > > > > > > have the
> > > > > > > >     > building blocks in place by the weekend so we can
> > > > > > > > start
> > > > working
> > > > > > > with
> > > > > > > > Infra
> > > > > > > >     > to get it in place.
> > > > > > > >     >
> > > > > > > >     > We will have a web server which we can point something
> > like
> > > > > > > >     > downloads.cloudstack.org (or whatever url) at.  Then
> > > > > > > > we
> > > will
> > > > > > have
> > > > > > > > an ASF
> > > > > > > >     > repo which tracks a mirror list and exposes a static
> > > > > > > > site
> > > to
> > > > be
> > > > > > > able
> > > > > > > > to
> > > > > > > >     > query the mirror list.
> > > > > > > >     >
> > > > > > > >     > The web server will receive a request and will do a
> > > > > > > > 302
> > > > > redirect
> > > > > > to
> > > > > > > > the
> > > > > > > >     > appropriate resource on one of the mirrors.
> > > > > > > >     >
> > > > > > > >     > This gets us started.  Legacy environments will have
> > > > > > > > to do
> > > a
> > > > DB
> > > > > > > > change to
> > > > > > > >     > change from 'download.cloud.com' to '
> > > > downloads.cloudstack.org'
> > > > > > (or
> > > > > > > >     > whatever), but otherwise it should be pretty simple.
> > > > > > > >     >
> > > > > > > >     > We can improve how we deliver templates going forward,
> > > > > > > > but
> > > > this
> > > > > > > > obviously
> > > > > > > >     > requires some discussion still.  I will try to buy us
> > > > > > > > some
> > > > time
> > > > > > > with
> > > > > > > > an
> > > > > > > >     > implementation which solves our problems today...
> > > > > > > >     >
> > > > > > > >     > *Will STEVENS*
> > > > > > > >     > Lead Developer
> > > > > > > >     >
> > > > > > > >     > <https://goo.gl/NYZ8KK>
> > > > > > > >     >
> > > > > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > > > > > > >     > abhinandan.prateek@shapeblue.com> wrote:
> > > > > > > >     >
> > > > > > > >     >>
> > > > > > > >     >> Initial seeding is a manual step and that is not
> > > > > > > > going to
> > > > > change
> > > > > > > in
> > > > > > > > near
> > > > > > > >     >> future. A handy list of official places from where
> > > > > > > > these
> > > > > > templates
> > > > > > > > can be
> > > > > > > >     >> downloaded won’t harm this part of the installation.
> > > > > > > > Just
> > > > > allow
> > > > > > > > anyone
> > > > > > > >     >> credible who follows democratic process to publish
> > > > > > > > their
> > > > > > templates
> > > > > > > > with
> > > > > > > >     >> apache’s blessings.
> > > > > > > >     >>
> > > > > > > >     >>
> > > > > > > >     >> Coming to urls hard coded in db, the popular
> > > > > > > > suggestion is
> > > > to
> > > > > > > point
> > > > > > > > them
> > > > > > > >     >> to mirrors. This looks good but this is going to take
> > > > > > > > some
> > > > > > effort
> > > > > > > > that has
> > > > > > > >     >> not materialised yet. Once this change is made it
> > > > > > > > will be
> > > > > going
> > > > > > in
> > > > > > > > some
> > > > > > > >     >> future release like 4.11; the problem still remains
> > > > > > > > for
> > > the
> > > > > > > > releases that
> > > > > > > >     >> are already out there. In case access to
> > > download.cloud.com
> > > > > is
> > > > > > > > dropped
> > > > > > > >     >> at some point we will have to document the procedure
> > > > > > > > such
> > > > that
> > > > > > > > people know
> > > > > > > >     >> how to make these stable releases work. That is where
> > > > > > > > the
> > > > > > > procedure
> > > > > > > >     >> documented here https://shankerbalan.net/blog/
> > > > > > > >     >> seed-cloudstack-templates-offline/ should be
> > > > > > > > officially
> > > > > adapted
> > > > > > > > with a
> > > > > > > >     >> marketplace to pick up templates of choosing. Maybe
> > > > > > > > we can
> > > > > have
> > > > > > a
> > > > > > > > pre-setup
> > > > > > > >     >> script to make choices instead of written wiki. Note
> > > > > > > > that
> > > > this
> > > > > > > will
> > > > > > > > be
> > > > > > > >     >> anyway needed by the older releases.
> > > > > > > >     >>
> > > > > > > >     >>
> > > > > > > >     >>
> > > > > > > >     >>
> > > > > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
> > > > paul.angus@shapeblue.com>
> > > > > > > > wrote:
> > > > > > > >     >>
> > > > > > > >     >> >-1
> > > > > > > >     >> >
> > > > > > > >     >> >We are trying to get MORE people to use CloudStack.
> > > > > > > > And
> > > > > > raising
> > > > > > > > the
> > > > > > > >     >> barrier to entry does nothing to help that.
> > > > > > > >     >> >
> > > > > > > >     >> >
> > > > > > > >     >> >Kind regards,
> > > > > > > >     >> >
> > > > > > > >     >> >Paul Angus
> > > > > > > >     >> >
> > > > > > > >     >> >paul.angus@shapeblue.com
> > > > > > > >     >> >www.shapeblue.com
> > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > >     >> >@shapeblue
> > > > > > > >     >> >
> > > > > > > >     >> >
> > > > > > > >     >> >
> > > > > > > >     >> >
> > > > > > > >     >> >-----Original Message-----
> > > > > > > >     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@
> > > > > > > shapeblue.com]
> > > > > > > >     >> >Sent: 01 March 2017 09:39
> > > > > > > >     >> >To: dev@cloudstack.apache.org
> > > > > > > >     >> >Subject: Re: Modern template hosting
> > > > > > > >     >> >
> > > > > > > >     >> >+1
> > > > > > > >     >> >
> > > > > > > >     >> >After seeing this discussion running in circles
> > > > > > > > several
> > > > > times,
> > > > > > I
> > > > > > > > think
> > > > > > > >     >> we should at least get started with the simplest
> option.
> > > > > > > >     >> >
> > > > > > > >     >> >
> > > > > > > >     >> >
> > > > > > > >     >> >
> > > > > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
> > > > > > chiradeepv@gmail.com>
> > > > > > > > wrote:
> > > > > > > >     >> >
> > > > > > > >     >> >>The alternative foolproof way is to simply not
> > > > > > > > provide
> > > the
> > > > > > > > automatic
> > > > > > > >     >> >>download.
> > > > > > > >     >> >>
> > > > > > > >     >> >>Just document the mirror list URL where the
> > > > > > > > templates
> > > can
> > > > be
> > > > > > > > found. I
> > > > > > > >     >> >>don't know why people reject this option.
> > > > > > > >     >> >>
> > > > > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
> > > > > > > templates-offline/
> > > > > > > >     >> >>
> > > > > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> > > > > > > >     >> >><wi...@gmail.com>
> > > > > > > >     >> >>wrote:
> > > > > > > >     >> >>
> > > > > > > >     >> >>> OK. Thanks for the heads up.
> > > > > > > >     >> >>>
> > > > > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > > > > > chiradeepv@gmail.com>
> > > > > > > >     >> wrote:
> > > > > > > >     >> >>>
> > > > > > > >     >> >>> > Sounds workable. The downloader code in the
> > > > > > > > SSVM
> > > won't
> > > > > > > follow
> > > > > > > >     >> >>> > redirects I think.
> > > > > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
> > > > > > > 5511065fc20787619d
> > > > > > > > 9cd0444
> > > > > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/
> > > storage/template/
> > > > > > > >     >> >>> > HttpTemplateDownloader.java#L93
> > > > > > > >     >> >>> > https://goo.gl/dSi0r5
> > > > > > > >     >> >>> >  Might need to add
> > > > > > > >     >> >>> > client.setRedirectStrategy(new
> > > LaxRedirectStrategy());
> > > > > > > >     >> >>> >
> > > > > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> > > > > > > >     >> >>> > <ws...@cloudops.com>
> > > > > > > >     >> >>> > wrote:
> > > > > > > >     >> >>> >
> > > > > > > >     >> >>> > > We haven't opened a ticket yet because we
> > > > > > > > don't
> > > > have a
> > > > > > > > strategy
> > > > > > > >     >> yet.
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> > > What do you guys think of this:
> > > > > > > >     >> >>> > > - We setup a new github repo in the 'apache'
> > > > > > > > org
> > > > which
> > > > > > > > consists
> > > > > > > >     >> >>> > > of a
> > > > > > > >     >> >>> > single
> > > > > > > >     >> >>> > > file with a list of active/supported mirrors.
> > > > > > > >     >> >>> > > - I write a small web server, distributed as
> > > > > > > > a
> > > > binary,
> > > > > > > > which can
> > > > > > > >     >> >>> > > be
> > > > > > > >     >> >>> > hosted
> > > > > > > >     >> >>> > > by ASF Infra.  This web server will query the
> > > > current
> > > > > > list
> > > > > > > > of
> > > > > > > >     >> >>> > > mirrors
> > > > > > > >     >> >>> and
> > > > > > > >     >> >>> > > will select one and then do a 302 redirect to
> > > > > > > > that
> > > > > > mirror.
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> > > The act of 'choosing' a mirror could be done
> > > > > > > > in a
> > > > > number
> > > > > > > of
> > > > > > > > ways.
> > > > > > > >     >> >>> > > - If we want to define an order, then it
> > > > > > > > could
> > > just
> > > > > try
> > > > > > > > from the
> > > > > > > >     >> >>> > > top of
> > > > > > > >     >> >>> > the
> > > > > > > >     >> >>> > > list and work its way down.  It would curl
> > > > > > > > the
> > > > target
> > > > > to
> > > > > > > > make
> > > > > > > >     >> >>> > > sure it
> > > > > > > >     >> >>> > gets
> > > > > > > >     >> >>> > > a 200 and if it does, it would do a 302
> > redirect.
> > > > > > > >     >> >>> > > - Or, if we want to distribute the load
> > > > > > > > across the
> > > > > > > mirrors,
> > > > > > > > we
> > > > > > > >     >> >>> > > could
> > > > > > > >     >> >>> pick
> > > > > > > >     >> >>> > > from the list randomly.  Again, doing a curl
> > > > > > > > to
> > > > verify
> > > > > > the
> > > > > > > > mirror
> > > > > > > >     >> >>> > > is up
> > > > > > > >     >> >>> > and
> > > > > > > >     >> >>> > > then doing a redirect.
> > > > > > > >     >> >>> > > - If we want to get fancy, we could do a
> > > > > > > > reverse
> > > IP
> > > > > > lookup
> > > > > > > > and
> > > > > > > >     >> >>> > > try to
> > > > > > > >     >> >>> > match
> > > > > > > >     >> >>> > > the requester with their closest geographical
> > > > mirror.
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> > > Thoughts?
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> > > *Will STEVENS*
> > > > > > > >     >> >>> > > Lead Developer
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep
> > > Vittal <
> > > > > > > >     >> >>> chiradeepv@gmail.com
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> > > wrote:
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> > > > What steps are needed to set up a mirror?
> > > > > > > > What
> > > > does
> > > > > > > Infra
> > > > > > > > need
> > > > > > > >     >> to do?
> > > > > > > >     >> >>> > Has
> > > > > > > >     >> >>> > > > anybody filed a ticket with Infra?
> > > > > > > >     >> >>> > > >
> > > > > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja
> > Pullela <
> > > > > > > >     >> >>> > > > raja.pullela@accelerite.com>
> > > > > > > >     >> >>> > > > wrote:
> > > > > > > >     >> >>> > > >
> > > > > > > >     >> >>> > > > > Hi will,
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > I believe, we didn’t get to close
> > > > > > > > ‘getting a
> > > > > mirror
> > > > > > on
> > > > > > > > Apache’
> > > > > > > >     >> >>> > because
> > > > > > > >     >> >>> > > we
> > > > > > > >     >> >>> > > > > needed someone on the Apache Infra side
> > > > > > > > to
> > > close
> > > > > > this.
> > > > > > > > BTW,
> > > > > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux
> > > > > > > > manages
> > > > this?)
> > > > > > has
> > > > > > > >     >> >>> > > > > all/most of
> > > > > > > >     >> >>> > the
> > > > > > > >     >> >>> > > > > content.  Once we can close on the Apache
> > > mirror
> > > > > for
> > > > > > > > hosting
> > > > > > > >     >> >>> > > > > the
> > > > > > > >     >> >>> > > > content, I
> > > > > > > >     >> >>> > > > > can help assist getting the content there.
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > For now, we have replicated the
> > > > > download.cloud.com
> > > > > > > > content to
> > > > > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
> > > > > > > >     >> >>> > > > > Also, we are working on a set of
> > > steps/procedure
> > > > > to
> > > > > > > > help with
> > > > > > > >     >> >>> > > > > this change.  I will update everyone in
> > > > > > > > about
> > > a
> > > > > > week’s
> > > > > > > > time
> > > > > > > >     >> >>> > > > > on the
> > > > > > > >     >> >>> > details.
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > Best,
> > > > > > > >     >> >>> > > > > Raja Pullela
> > > > > > > >     >> >>> > > > > Engineering Team,
> > > > > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa
> > > > > > > > Clara,
> > > > CA,
> > > > > > > 95054
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "
> > > williamstevens@gmail.com
> > > > > on
> > > > > > > > behalf of
> > > > > > > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com
> > > > > > > > on
> > > > behalf
> > > > > > of
> > > > > > > >     >> >>> > wstevens@cloudops.com>
> > > > > > > >     >> >>> > > > > wrote:
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > unfortunately the template mirror
> > > > > > > > conversation
> > > > got
> > > > > > > > caught up
> > > > > > > >     >> >>> > > > > in
> > > > > > > >     >> >>> > details
> > > > > > > >     >> >>> > > > and
> > > > > > > >     >> >>> > > > > nobody took the lead on implementing a
> > > solution.
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > citrix has been pinging me every couple
> > > > > > > > months
> > > > to
> > > > > > say
> > > > > > > > 'dude,
> > > > > > > >     >> >>> > > > > we
> > > > > > > >     >> >>> need
> > > > > > > >     >> >>> > to
> > > > > > > >     >> >>> > > > > remove the dependency on
> > > > > > > > download.citrix.com
> > > ',
> > > > > but
> > > > > > i
> > > > > > > > have not
> > > > > > > >     >> >>> > > > > had
> > > > > > > >     >> >>> > the
> > > > > > > >     >> >>> > > > > cycles to get in and solve the problem.
> > > > > > > > the
> > > > > > shutdown
> > > > > > > > of that
> > > > > > > >     >> >>> > > > > is
> > > > > > > >     >> >>> > > imminent
> > > > > > > >     >> >>> > > > > right now, so we need to solve it asap.
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > *Will STEVENS*
> > > > > > > >     >> >>> > > > > Lead Developer
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul
> > Angus <
> > > > > > > >     >> >>> > paul.angus@shapeblue.com
> > > > > > > >     >> >>> > > >
> > > > > > > >     >> >>> > > > > wrote:
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > > Hi Nathan,
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > Ideally, if you put the template
> > > > > > > > location in
> > > > (or
> > > > > > use
> > > > > > > a
> > > > > > > >     >> >>> > > > > > template
> > > > > > > >     >> >>> > > defined
> > > > > > > >     >> >>> > > > > > in)  test_data.py then the actual
> > > > > > > > location
> > > can
> > > > > be
> > > > > > > >     >> >>> > > > > > overridden by
> > > > > > > >     >> >>> > > anyone
> > > > > > > >     >> >>> > > > > > testing.
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > For Trillian, we've copied all of the
> > > > templates
> > > > > > that
> > > > > > > > people
> > > > > > > >     >> >>> > > > > > have
> > > > > > > >     >> >>> > > define
> > > > > > > >     >> >>> > > > > to
> > > > > > > >     >> >>> > > > > > a local repo and then replace the URLs
> > > > > > > > in test_data.py to
> > > > > > > >     >> >>> > > > > > reduce
> > > > > > > >     >> >>> > > > > bandwidth
> > > > > > > >     >> >>> > > > > > use and download times.
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > Ie:
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >             "bootableIso":
> > > > > > > >     >> >>> > > > > >                 {
> > > > > > > >     >> >>> > > > > >                     "displaytext": "Test
> > > > > Bootable
> > > > > > > > ISO",
> > > > > > > >     >> >>> > > > > >                     "name": "testISO",
> > > > > > > >     >> >>> > > > > >                     "bootable": True,
> > > > > > > >     >> >>> > > > > >                     "ispublic": False,
> > > > > > > >     >> >>> > > > > >                     "url": "{{
> > > > > > > marvin_images_location
> > > > > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
> > > > > > > >     >> >>> > > > > >                     "ostype": 'Other
> Linux
> > > > > > > (64-bit)',
> > > > > > > >     >> >>> > > > > >                     "mode":
> > 'HTTP_DOWNLOAD'
> > > > > > > >     >> >>> > > > > >         },
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > I thought that we had come up with a
> > > solution
> > > > > for
> > > > > > > >     >> >>> > download.cloud.com
> > > > > > > >     >> >>> > > ,
> > > > > > > >     >> >>> > > > by
> > > > > > > >     >> >>> > > > > > having a mirrorlist hosted in Community
> > > Apache
> > > > > > > > 'space' with
> > > > > > > >     >> >>> anyone
> > > > > > > >     >> >>> > > able
> > > > > > > >     >> >>> > > > > to
> > > > > > > >     >> >>> > > > > > out themselves forward as a mirror.
> > > > > > > >     >> >>> > > > > > But I must admit I lost track of
> > > > > > > > whether
> > > > anyone
> > > > > > made
> > > > > > > > the
> > > > > > > >     >> >>> requisite
> > > > > > > >     >> >>> > > > > changes
> > > > > > > >     >> >>> > > > > > in code....
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > Kind regards,
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > Paul Angus
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > paul.angus@shapeblue.com
> > > > > > > >     >> >>> > > > > > www.shapeblue.com
> > > > > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden, London
> > > WC2N
> > > > > > 4HSUK
> > > > > > > >     >> >>> > > > > > @shapeblue
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > -----Original Message-----
> > > > > > > >     >> >>> > > > > > From: williamstevens@gmail.com
> > > > > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
> > > > > > > >     >> >>> > On
> > > > > > > >     >> >>> > > > > > Behalf Of Will Stevens
> > > > > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
> > > > > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
> > > > > > > >     >> >>> > > > > > Subject: Re: Modern template hosting
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > this is a hard questions.  in general,
> > > > > > > > we
> > > > should
> > > > > > be
> > > > > > > > setting
> > > > > > > >     >> >>> > > > > > up a
> > > > > > > >     >> >>> > > mirror
> > > > > > > >     >> >>> > > > > on
> > > > > > > >     >> >>> > > > > > some cloudstack/apache domain and then
> > > mirror
> > > > to
> > > > > > > other
> > > > > > > >     >> >>> > > > > > provided
> > > > > > > >     >> >>> > > > > templates.
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > we MUST come up with a solution to
> > deprecate
> > > > > > > >     >> >>> > > > > > 'download.cloud.com
> > > > > > > >     >> >>> ',
> > > > > > > >     >> >>> > > > that
> > > > > > > >     >> >>> > > > > > is going to be going away any day now.
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > i don't know the right way to solve
> > > > > > > > this to
> > > be
> > > > > > > > honest, but
> > > > > > > >     >> >>> > > > > > if you
> > > > > > > >     >> >>> > > have
> > > > > > > >     >> >>> > > > > > ideas, i am willing to help.
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > *Will STEVENS*
> > > > > > > >     >> >>> > > > > > Lead Developer
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM,
> > > > > > > > Nathan
> > > > > Johnson <
> > > > > > > >     >> >>> njohnson@ena.com
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> > > > > wrote:
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > > > > So not to re-open a can of worms, but
> > > > > > > > I’m
> > > > in a
> > > > > > > > situation
> > > > > > > >     >> >>> > > > > > > where
> > > > > > > >     >> >>> I
> > > > > > > >     >> >>> > > need
> > > > > > > >     >> >>> > > > > > > to come up with a Marvin component
> > > > > > > > test
> > > that
> > > > > > > > depends on a
> > > > > > > >     >> >>> > template
> > > > > > > >     >> >>> > > > > > > based on a kernel that’s relatively
> > > > > > > > new,
> > > > i.e.,
> > > > > > > > newer than
> > > > > > > >     >> >>> Centos
> > > > > > > >     >> >>> > > 5.3
> > > > > > > >     >> >>> > > > /
> > > > > > > >     >> >>> > > > > > Ubuntu 10.04 .
> > > > > > > >     >> >>> > > > > > > I see openvm.eu has a suitable
> > > > > > > > template
> > > > > (Ubuntu
> > > > > > > > 16.0.4
> > > > > > > >     >> >>> > > > > > > for
> > > > > > > >     >> >>> KVM),
> > > > > > > >     >> >>> > > but
> > > > > > > >     >> >>> > > > > > > from looking at the thread "Migrating
> > > > > CloudStack
> > > > > > > > content
> > > > > > > >     >> >>> > > > > > > from download.cloud.com” it looks
> > > > > > > > like
> > > > there
> > > > > is
> > > > > > > >     >> >>> > > > > > > resistance to using
> > > > > > > >     >> >>> > > this
> > > > > > > >     >> >>> > > > at
> > > > > > > >     >> >>> > > > > > > least for hosting system vm templates
> > > > > > > > over
> > > > > > > concerns
> > > > > > > > of
> > > > > > > >     >> >>> > neutrality.
> > > > > > > >     >> >>> > > > > > > Would this be suitable for a
> > > > > > > > component
> > > test?
> > > > > If
> > > > > > > > not,
> > > > > > > >     >> >>> > > > > > > what is a
> > > > > > > >     >> >>> > > > > “blessed”
> > > > > > > >     >> >>> > > > > > > template location?
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > > Thanks in advance!
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > > Nathan Johnson
> > > > > > > >     >> >>> > > > > > > R&D Engineer
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > Nashville,
> > > > > TN
> > > > > > > > 37211
> > > > > > > >     >> >>> > > > > > > General Office: 615-312-6000
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > > website | blog | support
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > > >
> > > > > > > >     >> >>> > > > > >
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > > > DISCLAIMER
> > > > > > > >     >> >>> > > > > ==========
> > > > > > > >     >> >>> > > > > This e-mail may contain privileged and
> > > > > confidential
> > > > > > > >     >> >>> > > > > information
> > > > > > > >     >> >>> which
> > > > > > > >     >> >>> > > is
> > > > > > > >     >> >>> > > > > the property of Accelerite, a Persistent
> > > Systems
> > > > > > > > business. It
> > > > > > > >     >> >>> > > > > is
> > > > > > > >     >> >>> > > intended
> > > > > > > >     >> >>> > > > > only for the use of the individual or
> > > > > > > > entity
> > > to
> > > > > > which
> > > > > > > > it is
> > > > > > > >     >> >>> > addressed.
> > > > > > > >     >> >>> > > If
> > > > > > > >     >> >>> > > > > you are not the intended recipient, you
> > > > > > > > are
> > > not
> > > > > > > > authorized to
> > > > > > > >     >> >>> > > > > read,
> > > > > > > >     >> >>> > > > retain,
> > > > > > > >     >> >>> > > > > copy, print, distribute or use this
> message.
> > > If
> > > > > you
> > > > > > > have
> > > > > > > >     >> >>> > > > > received
> > > > > > > >     >> >>> > this
> > > > > > > >     >> >>> > > > > communication in error, please notify the
> > > sender
> > > > > and
> > > > > > > > delete
> > > > > > > >     >> >>> > > > > all
> > > > > > > >     >> >>> > copies
> > > > > > > >     >> >>> > > of
> > > > > > > >     >> >>> > > > > this message. Accelerite, a Persistent
> > > > > > > > Systems
> > > > > > > business
> > > > > > > > does
> > > > > > > >     >> >>> > > > > not
> > > > > > > >     >> >>> > accept
> > > > > > > >     >> >>> > > > any
> > > > > > > >     >> >>> > > > > liability for virus infected mails.
> > > > > > > >     >> >>> > > > >
> > > > > > > >     >> >>> > > >
> > > > > > > >     >> >>> > >
> > > > > > > >     >> >>> >
> > > > > > > >     >> >>>
> > > > > > > >     >> >
> > > > > > > >     >> >abhinandan.prateek@shapeblue.com
> > > > > > > >     >> >www.shapeblue.com
> > > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > @shapeblue
> > > > > > > >     >> >
> > > > > > > >     >> >
> > > > > > > >     >> >
> > > > > > > >     >>
> > > > > > > >     >> abhinandan.prateek@shapeblue.com
> > > > > > > >     >> www.shapeblue.com
> > > > > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > >     >> @shapeblue
> > > > > > > >     >>
> > > > > > > >     >>
> > > > > > > >     >>
> > > > > > > >     >>
> > > > > > > >     >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > daan.hoogland@shapeblue.com
> > > > > > > > www.shapeblue.com
> > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > > @shapeblue
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Rafael Weingärtner
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Rafael Weingärtner
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Modern template hosting

Posted by Chiradeep Vittal <ch...@gmail.com>.

Seriously?
apt-get install apache2

To install CloudStack, they need to know
- DB installation, perhaps some SQL
- VLAN configuration on their switches
- Ins-and-outs between port forwarding, static ips,
- NFS
- package management
- VHDs, qcow2. vmdk
- hyperviosrs
- and on and on.

If they can figure all that out and not figure out a web server, they can
always come to the mailing list.




On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <pa...@shapeblue.com> wrote:

> The issue is not with supporting highly experienced cloud operators. It is
> an issue for new users and other 'relatively' inexperienced operators.
>
> I have helped enough newbies and cloud operators who have been running
> their cloud for a while to know that the barriers to entry are too high as
> they are. And telling anyone that they need to create a web server so that
> they can add their initial template to get started or in order to create a
> new zone, just isn't going to fly.
>
> I'm a huge advocate of the 'download.cloudstack.org' endpoint which the
> community can add/remove mirrors to or from for system VMs or built-in
> templates.  Can the same system be used for binary repos ? although there
> is an added complication of redist vs no-redist there...
>
>
>
>
>
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: Chiradeep Vittal [mailto:chiradeepv@gmail.com]
> Sent: 03 March 2017 18:28
> To: dev <de...@cloudstack.apache.org>
> Subject: Re: Modern template hosting
>
> I do feel like this is early optimization. Mirrors rarely fail. I'd expect
> a single web server hosted on Apache Infra without any monitors to fail
> more often than a mirror. We already expect Wido's systemvm repository to
> be up all the time. And it has been. Similarly, I don't believe Nux's
> repository has ever been down. And if Accelerite wants to host on S3, that
> one is pretty solid as well.
>
> This is an infrequent operation in a cloud. After the cloud is installed,
> the download servers are only needed for a new zone. If we trust the user
> to run a cloud, surely he/she can run a web server to serve some built-in
> templates. And if her cloud is successful, she needs to figure out how to
> host her templates anyway and not rely on 3rd parties.
>
>
> On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <ws...@cloudops.com>
> wrote:
>
> > 1) If the legacy implementations do not support redirects, that does
> > cause a problem.  A potential solution in that case is to have the web
> > server actually proxy the download, but that is not ideal and I would
> > like to avoid it if possible.  Thanks for bringing that up Chiradeep.
> >
> > 2) I think we need to have a single URL which people can target.  Once
> > they make the switch to the new URL, we want the implementation to be
> > able to handle mirror failures without affecting the end client.  We
> > want to avoid the situation where an ACS user will ever have to change
> > this URL more than once.  Mirror failures SHOULD NOT affect the ACS
> > users assuming there is still at least one mirror who can serve the
> requested resource.
> >
> > These are obviously my personal opinions and others will probably have
> > differing opinions.
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal
> > <ch...@gmail.com>
> > wrote:
> >
> > > 1. If you are targeting legacy installations, they are not able to
> > > follow redirects. The line of code that added this capability was
> > > added on
> > 11/16.
> > > 2. If you trust the users to edit the database to change the URL,
> > > you can trust them to change it to anything. Just document a known
> > > list of good template locations.
> > >
> > > On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <ws...@cloudops.com>
> > > wrote:
> > >
> > > > So the main issue I see with this is the following.
> > > >
> > > > This implementation is designed to target legacy installations
> > > > which
> > will
> > > > be affected when download.cloud.com disappears.  These people will
> > need
> > > to
> > > > make a DB change to replace the 'download.cloud.com' with some
> > > > other
> > url
> > > > (maybe 'download.cloudstack.org').  Once the DB has been updated,
> > > > we
> > can
> > > > not expect anything else of the client, they need to be able to
> > > > just continue operation as they were without any need to rebuild
> > > > or upgrade their ACS.
> > > >
> > > > If we try to force the decision to the client, then we break the
> > > > legacy implementations.  Since the SSVM is likely going to be the
> > > > client in
> > some
> > > > cases, and since it does not already have the logic to handle the
> > > > 300 approach correctly, I feel like it is not a viable solution
> > > > for the
> > > legacy
> > > > deployments.
> > > >
> > > > In addition to that.  I don't think the client is in any better
> > position
> > > to
> > > > make the mirror decision than I am.  I am likely able to make a
> > > > more 'educated' decision than the client would be able to because
> > > > I can do different tests on the endpoint(s) before making a final
> decision.
> > > >
> > > > Does this make sense or am I missing something obvious?
> > > >
> > > > *Will STEVENS*
> > > > Lead Developer
> > > >
> > > > <https://goo.gl/NYZ8KK>
> > > >
> > > > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
> > > > rafaelweingartner@gmail.com> wrote:
> > > >
> > > > > I think I understood you, but I did not understand you.
> > > > >
> > > > > Let me see if I can create an example to illustrate how we could
> > > > > do
> > > using
> > > > > the HTTP 300 code.
> > > > >
> > > > > The HTTP 300 code indicates clients that he/she/it has multiple
> > choices
> > > > to
> > > > > access the requested resource. If the request is a HEAD, this is
> > > > > the
> > > only
> > > > > thing that the server returns. If the request if a GET, then the
> > server
> > > > > returns the choices for the client. The response type is defined
> > > > > on
> > the
> > > > > server based on the Content-Type, user agent and others.
> > > > >
> > > > > Let’s say the server receives a GET request and informs the
> > > content-type
> > > > as
> > > > > JSON (application/JSON). Your application could get the mirrors
> > > > > list,
> > > do
> > > > > the appending process for the requested resource path, and then
> > return
> > > a
> > > > > JSON array with the possible mirror the client can use. Then,
> > > > > the
> > > client
> > > > > gets this list and does the processing required to select a mirror.
> > > > >
> > > > > Now let’s say that a human access the link using a browser.
> > > > > Then, the server should receive something like (text/plain) as
> > > > > the
> > content-type;
> > > I
> > > > > think depending on the browser this field may differ a little
> > > > > (it
> > would
> > > > > require some checking). Anyways, the server detects that it is a
> > > “human”
> > > > > requesting the resource, then we could serve an HTML page with a
> > > > Javascript
> > > > > that uses the mirror list. This Javascript could do some
> > > > > checking and choose the best mirror for that specific.
> > > > >
> > > > > Why do I think this approach is interesting?
> > > > > The application you developed would be used only to retrieve
> > > > > valid
> > and
> > > > > trusted sources of system VM images (a trusted repository for
> > > > > mirrors
> > > > that
> > > > > we as a community vouch for).  Also, we shift the decision
> > > > > process regarding mirrors from the server to the client. Then,
> > > > > it is up to
> > > > clients
> > > > > to select mirrors, and not up to us (or some of our
> implementations).
> > > > >
> > > > > Did this help?
> > > > > BTW: I am not saying I am against the way you proposed, which
> > > > > would
> > > work
> > > > > fine. It is merely a suggestion using a different perspective.
> > > > >
> > > > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
> > > williamstevens@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > @rafael: in general, I don't think the client should ever be
> > making a
> > > > > > choice about a mirror. We have to assume we are working with a
> > > scripted
> > > > > > application and anything hard coded in that implementation on
> > > > > > the
> > > > client
> > > > > > side is a risk.
> > > > > >
> > > > > > I may not be understanding the use of the 300 approach, so I
> > > > > > will
> > > > > research
> > > > > > it to see if I can make it fit.
> > > > > >
> > > > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
> > > > > rafaelweingartner@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > Will great job.
> > > > > >
> > > > > > I had the same doubt as Daan.
> > > > > >
> > > > > > Have you considered using HTTP 300 working mode? Then, we
> > > > > > could let
> > > the
> > > > > > client decide which mirror is the best(closest?). Section
> “10.3.1”
> > of
> > > > [1]
> > > > > > talks about it; this would put the pressure on deciding upon a
> > mirror
> > > > on
> > > > > > the client side, but I think it would be fair.
> > > > > >
> > > > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
> > > > > >
> > > > > > PS: You really like very short variables names!
> > > > > >
> > > > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
> > > > > daan.hoogland@shapeblue.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Nice little thing Will,
> > > > > > >
> > > > > > > One question: if I read the code correctly it ‘go’es and
> > > > > > > tries
> > all
> > > > > > mirrors
> > > > > > > at once and whichever responses first is redirected to the
> > client.
> > > > This
> > > > > > > might well be the same every time. This might be the one
> > > > > > > closest
> > to
> > > > the
> > > > > > > mirror302 and not to the client. Is that correct or did I
> > > > > > > miss a
> > > > > weighing
> > > > > > > algorithm hidden in there?
> > > > > > >
> > > > > > > Good coding,
> > > > > > >
> > > > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
> > > > > > >
> > > > > > >     Hey All,
> > > > > > >     Please review this repo and tell me what you think:
> > > > > > >     https://github.com/swill/mirror302
> > > > > > >
> > > > > > >     Let me know if you have questions or you would like me
> > > > > > > to
> > make
> > > > > > changes
> > > > > > >     before I bring this topic up with ASF Infra and open a
> > > > > > > ticket
> > > to
> > > > > get
> > > > > > > this
> > > > > > >     implemented.
> > > > > > >
> > > > > > >     Cheers,
> > > > > > >
> > > > > > >     *Will Stevens*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> > > > > wstevens@cloudops.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >     > I am building a short term solution right now.  I
> > > > > > > hopefully
> > > > will
> > > > > > > have the
> > > > > > >     > building blocks in place by the weekend so we can
> > > > > > > start
> > > working
> > > > > > with
> > > > > > > Infra
> > > > > > >     > to get it in place.
> > > > > > >     >
> > > > > > >     > We will have a web server which we can point something
> like
> > > > > > >     > downloads.cloudstack.org (or whatever url) at.  Then
> > > > > > > we
> > will
> > > > > have
> > > > > > > an ASF
> > > > > > >     > repo which tracks a mirror list and exposes a static
> > > > > > > site
> > to
> > > be
> > > > > > able
> > > > > > > to
> > > > > > >     > query the mirror list.
> > > > > > >     >
> > > > > > >     > The web server will receive a request and will do a
> > > > > > > 302
> > > > redirect
> > > > > to
> > > > > > > the
> > > > > > >     > appropriate resource on one of the mirrors.
> > > > > > >     >
> > > > > > >     > This gets us started.  Legacy environments will have
> > > > > > > to do
> > a
> > > DB
> > > > > > > change to
> > > > > > >     > change from 'download.cloud.com' to '
> > > downloads.cloudstack.org'
> > > > > (or
> > > > > > >     > whatever), but otherwise it should be pretty simple.
> > > > > > >     >
> > > > > > >     > We can improve how we deliver templates going forward,
> > > > > > > but
> > > this
> > > > > > > obviously
> > > > > > >     > requires some discussion still.  I will try to buy us
> > > > > > > some
> > > time
> > > > > > with
> > > > > > > an
> > > > > > >     > implementation which solves our problems today...
> > > > > > >     >
> > > > > > >     > *Will STEVENS*
> > > > > > >     > Lead Developer
> > > > > > >     >
> > > > > > >     > <https://goo.gl/NYZ8KK>
> > > > > > >     >
> > > > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > > > > > >     > abhinandan.prateek@shapeblue.com> wrote:
> > > > > > >     >
> > > > > > >     >>
> > > > > > >     >> Initial seeding is a manual step and that is not
> > > > > > > going to
> > > > change
> > > > > > in
> > > > > > > near
> > > > > > >     >> future. A handy list of official places from where
> > > > > > > these
> > > > > templates
> > > > > > > can be
> > > > > > >     >> downloaded won’t harm this part of the installation.
> > > > > > > Just
> > > > allow
> > > > > > > anyone
> > > > > > >     >> credible who follows democratic process to publish
> > > > > > > their
> > > > > templates
> > > > > > > with
> > > > > > >     >> apache’s blessings.
> > > > > > >     >>
> > > > > > >     >>
> > > > > > >     >> Coming to urls hard coded in db, the popular
> > > > > > > suggestion is
> > > to
> > > > > > point
> > > > > > > them
> > > > > > >     >> to mirrors. This looks good but this is going to take
> > > > > > > some
> > > > > effort
> > > > > > > that has
> > > > > > >     >> not materialised yet. Once this change is made it
> > > > > > > will be
> > > > going
> > > > > in
> > > > > > > some
> > > > > > >     >> future release like 4.11; the problem still remains
> > > > > > > for
> > the
> > > > > > > releases that
> > > > > > >     >> are already out there. In case access to
> > download.cloud.com
> > > > is
> > > > > > > dropped
> > > > > > >     >> at some point we will have to document the procedure
> > > > > > > such
> > > that
> > > > > > > people know
> > > > > > >     >> how to make these stable releases work. That is where
> > > > > > > the
> > > > > > procedure
> > > > > > >     >> documented here https://shankerbalan.net/blog/
> > > > > > >     >> seed-cloudstack-templates-offline/ should be
> > > > > > > officially
> > > > adapted
> > > > > > > with a
> > > > > > >     >> marketplace to pick up templates of choosing. Maybe
> > > > > > > we can
> > > > have
> > > > > a
> > > > > > > pre-setup
> > > > > > >     >> script to make choices instead of written wiki. Note
> > > > > > > that
> > > this
> > > > > > will
> > > > > > > be
> > > > > > >     >> anyway needed by the older releases.
> > > > > > >     >>
> > > > > > >     >>
> > > > > > >     >>
> > > > > > >     >>
> > > > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
> > > paul.angus@shapeblue.com>
> > > > > > > wrote:
> > > > > > >     >>
> > > > > > >     >> >-1
> > > > > > >     >> >
> > > > > > >     >> >We are trying to get MORE people to use CloudStack.
> > > > > > > And
> > > > > raising
> > > > > > > the
> > > > > > >     >> barrier to entry does nothing to help that.
> > > > > > >     >> >
> > > > > > >     >> >
> > > > > > >     >> >Kind regards,
> > > > > > >     >> >
> > > > > > >     >> >Paul Angus
> > > > > > >     >> >
> > > > > > >     >> >paul.angus@shapeblue.com
> > > > > > >     >> >www.shapeblue.com
> > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > >     >> >@shapeblue
> > > > > > >     >> >
> > > > > > >     >> >
> > > > > > >     >> >
> > > > > > >     >> >
> > > > > > >     >> >-----Original Message-----
> > > > > > >     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@
> > > > > > shapeblue.com]
> > > > > > >     >> >Sent: 01 March 2017 09:39
> > > > > > >     >> >To: dev@cloudstack.apache.org
> > > > > > >     >> >Subject: Re: Modern template hosting
> > > > > > >     >> >
> > > > > > >     >> >+1
> > > > > > >     >> >
> > > > > > >     >> >After seeing this discussion running in circles
> > > > > > > several
> > > > times,
> > > > > I
> > > > > > > think
> > > > > > >     >> we should at least get started with the simplest option.
> > > > > > >     >> >
> > > > > > >     >> >
> > > > > > >     >> >
> > > > > > >     >> >
> > > > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
> > > > > chiradeepv@gmail.com>
> > > > > > > wrote:
> > > > > > >     >> >
> > > > > > >     >> >>The alternative foolproof way is to simply not
> > > > > > > provide
> > the
> > > > > > > automatic
> > > > > > >     >> >>download.
> > > > > > >     >> >>
> > > > > > >     >> >>Just document the mirror list URL where the
> > > > > > > templates
> > can
> > > be
> > > > > > > found. I
> > > > > > >     >> >>don't know why people reject this option.
> > > > > > >     >> >>
> > > > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
> > > > > > templates-offline/
> > > > > > >     >> >>
> > > > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> > > > > > >     >> >><wi...@gmail.com>
> > > > > > >     >> >>wrote:
> > > > > > >     >> >>
> > > > > > >     >> >>> OK. Thanks for the heads up.
> > > > > > >     >> >>>
> > > > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > > > > chiradeepv@gmail.com>
> > > > > > >     >> wrote:
> > > > > > >     >> >>>
> > > > > > >     >> >>> > Sounds workable. The downloader code in the
> > > > > > > SSVM
> > won't
> > > > > > follow
> > > > > > >     >> >>> > redirects I think.
> > > > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
> > > > > > 5511065fc20787619d
> > > > > > > 9cd0444
> > > > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/
> > storage/template/
> > > > > > >     >> >>> > HttpTemplateDownloader.java#L93
> > > > > > >     >> >>> > https://goo.gl/dSi0r5
> > > > > > >     >> >>> >  Might need to add
> > > > > > >     >> >>> > client.setRedirectStrategy(new
> > LaxRedirectStrategy());
> > > > > > >     >> >>> >
> > > > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> > > > > > >     >> >>> > <ws...@cloudops.com>
> > > > > > >     >> >>> > wrote:
> > > > > > >     >> >>> >
> > > > > > >     >> >>> > > We haven't opened a ticket yet because we
> > > > > > > don't
> > > have a
> > > > > > > strategy
> > > > > > >     >> yet.
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> > > What do you guys think of this:
> > > > > > >     >> >>> > > - We setup a new github repo in the 'apache'
> > > > > > > org
> > > which
> > > > > > > consists
> > > > > > >     >> >>> > > of a
> > > > > > >     >> >>> > single
> > > > > > >     >> >>> > > file with a list of active/supported mirrors.
> > > > > > >     >> >>> > > - I write a small web server, distributed as
> > > > > > > a
> > > binary,
> > > > > > > which can
> > > > > > >     >> >>> > > be
> > > > > > >     >> >>> > hosted
> > > > > > >     >> >>> > > by ASF Infra.  This web server will query the
> > > current
> > > > > list
> > > > > > > of
> > > > > > >     >> >>> > > mirrors
> > > > > > >     >> >>> and
> > > > > > >     >> >>> > > will select one and then do a 302 redirect to
> > > > > > > that
> > > > > mirror.
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> > > The act of 'choosing' a mirror could be done
> > > > > > > in a
> > > > number
> > > > > > of
> > > > > > > ways.
> > > > > > >     >> >>> > > - If we want to define an order, then it
> > > > > > > could
> > just
> > > > try
> > > > > > > from the
> > > > > > >     >> >>> > > top of
> > > > > > >     >> >>> > the
> > > > > > >     >> >>> > > list and work its way down.  It would curl
> > > > > > > the
> > > target
> > > > to
> > > > > > > make
> > > > > > >     >> >>> > > sure it
> > > > > > >     >> >>> > gets
> > > > > > >     >> >>> > > a 200 and if it does, it would do a 302
> redirect.
> > > > > > >     >> >>> > > - Or, if we want to distribute the load
> > > > > > > across the
> > > > > > mirrors,
> > > > > > > we
> > > > > > >     >> >>> > > could
> > > > > > >     >> >>> pick
> > > > > > >     >> >>> > > from the list randomly.  Again, doing a curl
> > > > > > > to
> > > verify
> > > > > the
> > > > > > > mirror
> > > > > > >     >> >>> > > is up
> > > > > > >     >> >>> > and
> > > > > > >     >> >>> > > then doing a redirect.
> > > > > > >     >> >>> > > - If we want to get fancy, we could do a
> > > > > > > reverse
> > IP
> > > > > lookup
> > > > > > > and
> > > > > > >     >> >>> > > try to
> > > > > > >     >> >>> > match
> > > > > > >     >> >>> > > the requester with their closest geographical
> > > mirror.
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> > > Thoughts?
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> > > *Will STEVENS*
> > > > > > >     >> >>> > > Lead Developer
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep
> > Vittal <
> > > > > > >     >> >>> chiradeepv@gmail.com
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> > > wrote:
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> > > > What steps are needed to set up a mirror?
> > > > > > > What
> > > does
> > > > > > Infra
> > > > > > > need
> > > > > > >     >> to do?
> > > > > > >     >> >>> > Has
> > > > > > >     >> >>> > > > anybody filed a ticket with Infra?
> > > > > > >     >> >>> > > >
> > > > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja
> Pullela <
> > > > > > >     >> >>> > > > raja.pullela@accelerite.com>
> > > > > > >     >> >>> > > > wrote:
> > > > > > >     >> >>> > > >
> > > > > > >     >> >>> > > > > Hi will,
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > I believe, we didn’t get to close
> > > > > > > ‘getting a
> > > > mirror
> > > > > on
> > > > > > > Apache’
> > > > > > >     >> >>> > because
> > > > > > >     >> >>> > > we
> > > > > > >     >> >>> > > > > needed someone on the Apache Infra side
> > > > > > > to
> > close
> > > > > this.
> > > > > > > BTW,
> > > > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux
> > > > > > > manages
> > > this?)
> > > > > has
> > > > > > >     >> >>> > > > > all/most of
> > > > > > >     >> >>> > the
> > > > > > >     >> >>> > > > > content.  Once we can close on the Apache
> > mirror
> > > > for
> > > > > > > hosting
> > > > > > >     >> >>> > > > > the
> > > > > > >     >> >>> > > > content, I
> > > > > > >     >> >>> > > > > can help assist getting the content there.
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > For now, we have replicated the
> > > > download.cloud.com
> > > > > > > content to
> > > > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
> > > > > > >     >> >>> > > > > Also, we are working on a set of
> > steps/procedure
> > > > to
> > > > > > > help with
> > > > > > >     >> >>> > > > > this change.  I will update everyone in
> > > > > > > about
> > a
> > > > > week’s
> > > > > > > time
> > > > > > >     >> >>> > > > > on the
> > > > > > >     >> >>> > details.
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > Best,
> > > > > > >     >> >>> > > > > Raja Pullela
> > > > > > >     >> >>> > > > > Engineering Team,
> > > > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa
> > > > > > > Clara,
> > > CA,
> > > > > > 95054
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "
> > williamstevens@gmail.com
> > > > on
> > > > > > > behalf of
> > > > > > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com
> > > > > > > on
> > > behalf
> > > > > of
> > > > > > >     >> >>> > wstevens@cloudops.com>
> > > > > > >     >> >>> > > > > wrote:
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > unfortunately the template mirror
> > > > > > > conversation
> > > got
> > > > > > > caught up
> > > > > > >     >> >>> > > > > in
> > > > > > >     >> >>> > details
> > > > > > >     >> >>> > > > and
> > > > > > >     >> >>> > > > > nobody took the lead on implementing a
> > solution.
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > citrix has been pinging me every couple
> > > > > > > months
> > > to
> > > > > say
> > > > > > > 'dude,
> > > > > > >     >> >>> > > > > we
> > > > > > >     >> >>> need
> > > > > > >     >> >>> > to
> > > > > > >     >> >>> > > > > remove the dependency on
> > > > > > > download.citrix.com
> > ',
> > > > but
> > > > > i
> > > > > > > have not
> > > > > > >     >> >>> > > > > had
> > > > > > >     >> >>> > the
> > > > > > >     >> >>> > > > > cycles to get in and solve the problem.
> > > > > > > the
> > > > > shutdown
> > > > > > > of that
> > > > > > >     >> >>> > > > > is
> > > > > > >     >> >>> > > imminent
> > > > > > >     >> >>> > > > > right now, so we need to solve it asap.
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > *Will STEVENS*
> > > > > > >     >> >>> > > > > Lead Developer
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul
> Angus <
> > > > > > >     >> >>> > paul.angus@shapeblue.com
> > > > > > >     >> >>> > > >
> > > > > > >     >> >>> > > > > wrote:
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > > Hi Nathan,
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > Ideally, if you put the template
> > > > > > > location in
> > > (or
> > > > > use
> > > > > > a
> > > > > > >     >> >>> > > > > > template
> > > > > > >     >> >>> > > defined
> > > > > > >     >> >>> > > > > > in)  test_data.py then the actual
> > > > > > > location
> > can
> > > > be
> > > > > > >     >> >>> > > > > > overridden by
> > > > > > >     >> >>> > > anyone
> > > > > > >     >> >>> > > > > > testing.
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > For Trillian, we've copied all of the
> > > templates
> > > > > that
> > > > > > > people
> > > > > > >     >> >>> > > > > > have
> > > > > > >     >> >>> > > define
> > > > > > >     >> >>> > > > > to
> > > > > > >     >> >>> > > > > > a local repo and then replace the URLs
> > > > > > > in test_data.py to
> > > > > > >     >> >>> > > > > > reduce
> > > > > > >     >> >>> > > > > bandwidth
> > > > > > >     >> >>> > > > > > use and download times.
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > Ie:
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >             "bootableIso":
> > > > > > >     >> >>> > > > > >                 {
> > > > > > >     >> >>> > > > > >                     "displaytext": "Test
> > > > Bootable
> > > > > > > ISO",
> > > > > > >     >> >>> > > > > >                     "name": "testISO",
> > > > > > >     >> >>> > > > > >                     "bootable": True,
> > > > > > >     >> >>> > > > > >                     "ispublic": False,
> > > > > > >     >> >>> > > > > >                     "url": "{{
> > > > > > marvin_images_location
> > > > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
> > > > > > >     >> >>> > > > > >                     "ostype": 'Other Linux
> > > > > > (64-bit)',
> > > > > > >     >> >>> > > > > >                     "mode":
> 'HTTP_DOWNLOAD'
> > > > > > >     >> >>> > > > > >         },
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > I thought that we had come up with a
> > solution
> > > > for
> > > > > > >     >> >>> > download.cloud.com
> > > > > > >     >> >>> > > ,
> > > > > > >     >> >>> > > > by
> > > > > > >     >> >>> > > > > > having a mirrorlist hosted in Community
> > Apache
> > > > > > > 'space' with
> > > > > > >     >> >>> anyone
> > > > > > >     >> >>> > > able
> > > > > > >     >> >>> > > > > to
> > > > > > >     >> >>> > > > > > out themselves forward as a mirror.
> > > > > > >     >> >>> > > > > > But I must admit I lost track of
> > > > > > > whether
> > > anyone
> > > > > made
> > > > > > > the
> > > > > > >     >> >>> requisite
> > > > > > >     >> >>> > > > > changes
> > > > > > >     >> >>> > > > > > in code....
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > Kind regards,
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > Paul Angus
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > paul.angus@shapeblue.com
> > > > > > >     >> >>> > > > > > www.shapeblue.com
> > > > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden, London
> > WC2N
> > > > > 4HSUK
> > > > > > >     >> >>> > > > > > @shapeblue
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > -----Original Message-----
> > > > > > >     >> >>> > > > > > From: williamstevens@gmail.com
> > > > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
> > > > > > >     >> >>> > On
> > > > > > >     >> >>> > > > > > Behalf Of Will Stevens
> > > > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
> > > > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
> > > > > > >     >> >>> > > > > > Subject: Re: Modern template hosting
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > this is a hard questions.  in general,
> > > > > > > we
> > > should
> > > > > be
> > > > > > > setting
> > > > > > >     >> >>> > > > > > up a
> > > > > > >     >> >>> > > mirror
> > > > > > >     >> >>> > > > > on
> > > > > > >     >> >>> > > > > > some cloudstack/apache domain and then
> > mirror
> > > to
> > > > > > other
> > > > > > >     >> >>> > > > > > provided
> > > > > > >     >> >>> > > > > templates.
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > we MUST come up with a solution to
> deprecate
> > > > > > >     >> >>> > > > > > 'download.cloud.com
> > > > > > >     >> >>> ',
> > > > > > >     >> >>> > > > that
> > > > > > >     >> >>> > > > > > is going to be going away any day now.
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > i don't know the right way to solve
> > > > > > > this to
> > be
> > > > > > > honest, but
> > > > > > >     >> >>> > > > > > if you
> > > > > > >     >> >>> > > have
> > > > > > >     >> >>> > > > > > ideas, i am willing to help.
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > *Will STEVENS*
> > > > > > >     >> >>> > > > > > Lead Developer
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM,
> > > > > > > Nathan
> > > > Johnson <
> > > > > > >     >> >>> njohnson@ena.com
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> > > > > wrote:
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > > > > So not to re-open a can of worms, but
> > > > > > > I’m
> > > in a
> > > > > > > situation
> > > > > > >     >> >>> > > > > > > where
> > > > > > >     >> >>> I
> > > > > > >     >> >>> > > need
> > > > > > >     >> >>> > > > > > > to come up with a Marvin component
> > > > > > > test
> > that
> > > > > > > depends on a
> > > > > > >     >> >>> > template
> > > > > > >     >> >>> > > > > > > based on a kernel that’s relatively
> > > > > > > new,
> > > i.e.,
> > > > > > > newer than
> > > > > > >     >> >>> Centos
> > > > > > >     >> >>> > > 5.3
> > > > > > >     >> >>> > > > /
> > > > > > >     >> >>> > > > > > Ubuntu 10.04 .
> > > > > > >     >> >>> > > > > > > I see openvm.eu has a suitable
> > > > > > > template
> > > > (Ubuntu
> > > > > > > 16.0.4
> > > > > > >     >> >>> > > > > > > for
> > > > > > >     >> >>> KVM),
> > > > > > >     >> >>> > > but
> > > > > > >     >> >>> > > > > > > from looking at the thread "Migrating
> > > > CloudStack
> > > > > > > content
> > > > > > >     >> >>> > > > > > > from download.cloud.com” it looks
> > > > > > > like
> > > there
> > > > is
> > > > > > >     >> >>> > > > > > > resistance to using
> > > > > > >     >> >>> > > this
> > > > > > >     >> >>> > > > at
> > > > > > >     >> >>> > > > > > > least for hosting system vm templates
> > > > > > > over
> > > > > > concerns
> > > > > > > of
> > > > > > >     >> >>> > neutrality.
> > > > > > >     >> >>> > > > > > > Would this be suitable for a
> > > > > > > component
> > test?
> > > > If
> > > > > > > not,
> > > > > > >     >> >>> > > > > > > what is a
> > > > > > >     >> >>> > > > > “blessed”
> > > > > > >     >> >>> > > > > > > template location?
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > > Thanks in advance!
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > > Nathan Johnson
> > > > > > >     >> >>> > > > > > > R&D Engineer
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12
> > > Nashville,
> > > > TN
> > > > > > > 37211
> > > > > > >     >> >>> > > > > > > General Office: 615-312-6000
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > > website | blog | support
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > > >
> > > > > > >     >> >>> > > > > >
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > > > DISCLAIMER
> > > > > > >     >> >>> > > > > ==========
> > > > > > >     >> >>> > > > > This e-mail may contain privileged and
> > > > confidential
> > > > > > >     >> >>> > > > > information
> > > > > > >     >> >>> which
> > > > > > >     >> >>> > > is
> > > > > > >     >> >>> > > > > the property of Accelerite, a Persistent
> > Systems
> > > > > > > business. It
> > > > > > >     >> >>> > > > > is
> > > > > > >     >> >>> > > intended
> > > > > > >     >> >>> > > > > only for the use of the individual or
> > > > > > > entity
> > to
> > > > > which
> > > > > > > it is
> > > > > > >     >> >>> > addressed.
> > > > > > >     >> >>> > > If
> > > > > > >     >> >>> > > > > you are not the intended recipient, you
> > > > > > > are
> > not
> > > > > > > authorized to
> > > > > > >     >> >>> > > > > read,
> > > > > > >     >> >>> > > > retain,
> > > > > > >     >> >>> > > > > copy, print, distribute or use this message.
> > If
> > > > you
> > > > > > have
> > > > > > >     >> >>> > > > > received
> > > > > > >     >> >>> > this
> > > > > > >     >> >>> > > > > communication in error, please notify the
> > sender
> > > > and
> > > > > > > delete
> > > > > > >     >> >>> > > > > all
> > > > > > >     >> >>> > copies
> > > > > > >     >> >>> > > of
> > > > > > >     >> >>> > > > > this message. Accelerite, a Persistent
> > > > > > > Systems
> > > > > > business
> > > > > > > does
> > > > > > >     >> >>> > > > > not
> > > > > > >     >> >>> > accept
> > > > > > >     >> >>> > > > any
> > > > > > >     >> >>> > > > > liability for virus infected mails.
> > > > > > >     >> >>> > > > >
> > > > > > >     >> >>> > > >
> > > > > > >     >> >>> > >
> > > > > > >     >> >>> >
> > > > > > >     >> >>>
> > > > > > >     >> >
> > > > > > >     >> >abhinandan.prateek@shapeblue.com
> > > > > > >     >> >www.shapeblue.com
> > > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > @shapeblue
> > > > > > >     >> >
> > > > > > >     >> >
> > > > > > >     >> >
> > > > > > >     >>
> > > > > > >     >> abhinandan.prateek@shapeblue.com
> > > > > > >     >> www.shapeblue.com
> > > > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > >     >> @shapeblue
> > > > > > >     >>
> > > > > > >     >>
> > > > > > >     >>
> > > > > > >     >>
> > > > > > >     >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > daan.hoogland@shapeblue.com
> > > > > > > www.shapeblue.com
> > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > @shapeblue
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Rafael Weingärtner
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rafael Weingärtner
> > > > >
> > > >
> > >
> >
>

RE: Modern template hosting

Posted by Paul Angus <pa...@shapeblue.com>.

The issue is not with supporting highly experienced cloud operators. It is an issue for new users and other 'relatively' inexperienced operators.

I have helped enough newbies and cloud operators who have been running their cloud for a while to know that the barriers to entry are too high as they are. And telling anyone that they need to create a web server so that they can add their initial template to get started or in order to create a new zone, just isn't going to fly.

I'm a huge advocate of the 'download.cloudstack.org' endpoint which the community can add/remove mirrors to or from for system VMs or built-in templates.  Can the same system be used for binary repos ? although there is an added complication of redist vs no-redist there...




 

paul.angus@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-----Original Message-----
From: Chiradeep Vittal [mailto:chiradeepv@gmail.com] 
Sent: 03 March 2017 18:28
To: dev <de...@cloudstack.apache.org>
Subject: Re: Modern template hosting

I do feel like this is early optimization. Mirrors rarely fail. I'd expect a single web server hosted on Apache Infra without any monitors to fail more often than a mirror. We already expect Wido's systemvm repository to be up all the time. And it has been. Similarly, I don't believe Nux's repository has ever been down. And if Accelerite wants to host on S3, that one is pretty solid as well.

This is an infrequent operation in a cloud. After the cloud is installed, the download servers are only needed for a new zone. If we trust the user to run a cloud, surely he/she can run a web server to serve some built-in templates. And if her cloud is successful, she needs to figure out how to host her templates anyway and not rely on 3rd parties.


On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <ws...@cloudops.com> wrote:

> 1) If the legacy implementations do not support redirects, that does 
> cause a problem.  A potential solution in that case is to have the web 
> server actually proxy the download, but that is not ideal and I would 
> like to avoid it if possible.  Thanks for bringing that up Chiradeep.
>
> 2) I think we need to have a single URL which people can target.  Once 
> they make the switch to the new URL, we want the implementation to be 
> able to handle mirror failures without affecting the end client.  We 
> want to avoid the situation where an ACS user will ever have to change 
> this URL more than once.  Mirror failures SHOULD NOT affect the ACS 
> users assuming there is still at least one mirror who can serve the requested resource.
>
> These are obviously my personal opinions and others will probably have 
> differing opinions.
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal 
> <ch...@gmail.com>
> wrote:
>
> > 1. If you are targeting legacy installations, they are not able to 
> > follow redirects. The line of code that added this capability was 
> > added on
> 11/16.
> > 2. If you trust the users to edit the database to change the URL, 
> > you can trust them to change it to anything. Just document a known 
> > list of good template locations.
> >
> > On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <ws...@cloudops.com>
> > wrote:
> >
> > > So the main issue I see with this is the following.
> > >
> > > This implementation is designed to target legacy installations 
> > > which
> will
> > > be affected when download.cloud.com disappears.  These people will
> need
> > to
> > > make a DB change to replace the 'download.cloud.com' with some 
> > > other
> url
> > > (maybe 'download.cloudstack.org').  Once the DB has been updated, 
> > > we
> can
> > > not expect anything else of the client, they need to be able to 
> > > just continue operation as they were without any need to rebuild 
> > > or upgrade their ACS.
> > >
> > > If we try to force the decision to the client, then we break the 
> > > legacy implementations.  Since the SSVM is likely going to be the 
> > > client in
> some
> > > cases, and since it does not already have the logic to handle the 
> > > 300 approach correctly, I feel like it is not a viable solution 
> > > for the
> > legacy
> > > deployments.
> > >
> > > In addition to that.  I don't think the client is in any better
> position
> > to
> > > make the mirror decision than I am.  I am likely able to make a 
> > > more 'educated' decision than the client would be able to because 
> > > I can do different tests on the endpoint(s) before making a final decision.
> > >
> > > Does this make sense or am I missing something obvious?
> > >
> > > *Will STEVENS*
> > > Lead Developer
> > >
> > > <https://goo.gl/NYZ8KK>
> > >
> > > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner < 
> > > rafaelweingartner@gmail.com> wrote:
> > >
> > > > I think I understood you, but I did not understand you.
> > > >
> > > > Let me see if I can create an example to illustrate how we could 
> > > > do
> > using
> > > > the HTTP 300 code.
> > > >
> > > > The HTTP 300 code indicates clients that he/she/it has multiple
> choices
> > > to
> > > > access the requested resource. If the request is a HEAD, this is 
> > > > the
> > only
> > > > thing that the server returns. If the request if a GET, then the
> server
> > > > returns the choices for the client. The response type is defined 
> > > > on
> the
> > > > server based on the Content-Type, user agent and others.
> > > >
> > > > Let’s say the server receives a GET request and informs the
> > content-type
> > > as
> > > > JSON (application/JSON). Your application could get the mirrors 
> > > > list,
> > do
> > > > the appending process for the requested resource path, and then
> return
> > a
> > > > JSON array with the possible mirror the client can use. Then, 
> > > > the
> > client
> > > > gets this list and does the processing required to select a mirror.
> > > >
> > > > Now let’s say that a human access the link using a browser. 
> > > > Then, the server should receive something like (text/plain) as 
> > > > the
> content-type;
> > I
> > > > think depending on the browser this field may differ a little 
> > > > (it
> would
> > > > require some checking). Anyways, the server detects that it is a
> > “human”
> > > > requesting the resource, then we could serve an HTML page with a
> > > Javascript
> > > > that uses the mirror list. This Javascript could do some 
> > > > checking and choose the best mirror for that specific.
> > > >
> > > > Why do I think this approach is interesting?
> > > > The application you developed would be used only to retrieve 
> > > > valid
> and
> > > > trusted sources of system VM images (a trusted repository for 
> > > > mirrors
> > > that
> > > > we as a community vouch for).  Also, we shift the decision 
> > > > process regarding mirrors from the server to the client. Then, 
> > > > it is up to
> > > clients
> > > > to select mirrors, and not up to us (or some of our implementations).
> > > >
> > > > Did this help?
> > > > BTW: I am not saying I am against the way you proposed, which 
> > > > would
> > work
> > > > fine. It is merely a suggestion using a different perspective.
> > > >
> > > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
> > williamstevens@gmail.com>
> > > > wrote:
> > > >
> > > > > @rafael: in general, I don't think the client should ever be
> making a
> > > > > choice about a mirror. We have to assume we are working with a
> > scripted
> > > > > application and anything hard coded in that implementation on 
> > > > > the
> > > client
> > > > > side is a risk.
> > > > >
> > > > > I may not be understanding the use of the 300 approach, so I 
> > > > > will
> > > > research
> > > > > it to see if I can make it fit.
> > > > >
> > > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
> > > > rafaelweingartner@gmail.com>
> > > > > wrote:
> > > > >
> > > > > Will great job.
> > > > >
> > > > > I had the same doubt as Daan.
> > > > >
> > > > > Have you considered using HTTP 300 working mode? Then, we 
> > > > > could let
> > the
> > > > > client decide which mirror is the best(closest?). Section “10.3.1”
> of
> > > [1]
> > > > > talks about it; this would put the pressure on deciding upon a
> mirror
> > > on
> > > > > the client side, but I think it would be fair.
> > > > >
> > > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
> > > > >
> > > > > PS: You really like very short variables names!
> > > > >
> > > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
> > > > daan.hoogland@shapeblue.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Nice little thing Will,
> > > > > >
> > > > > > One question: if I read the code correctly it ‘go’es and 
> > > > > > tries
> all
> > > > > mirrors
> > > > > > at once and whichever responses first is redirected to the
> client.
> > > This
> > > > > > might well be the same every time. This might be the one 
> > > > > > closest
> to
> > > the
> > > > > > mirror302 and not to the client. Is that correct or did I 
> > > > > > miss a
> > > > weighing
> > > > > > algorithm hidden in there?
> > > > > >
> > > > > > Good coding,
> > > > > >
> > > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
> > > > > >
> > > > > >     Hey All,
> > > > > >     Please review this repo and tell me what you think:
> > > > > >     https://github.com/swill/mirror302
> > > > > >
> > > > > >     Let me know if you have questions or you would like me 
> > > > > > to
> make
> > > > > changes
> > > > > >     before I bring this topic up with ASF Infra and open a 
> > > > > > ticket
> > to
> > > > get
> > > > > > this
> > > > > >     implemented.
> > > > > >
> > > > > >     Cheers,
> > > > > >
> > > > > >     *Will Stevens*
> > > > > >
> > > > > >
> > > > > >
> > > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> > > > wstevens@cloudops.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >     > I am building a short term solution right now.  I 
> > > > > > hopefully
> > > will
> > > > > > have the
> > > > > >     > building blocks in place by the weekend so we can 
> > > > > > start
> > working
> > > > > with
> > > > > > Infra
> > > > > >     > to get it in place.
> > > > > >     >
> > > > > >     > We will have a web server which we can point something like
> > > > > >     > downloads.cloudstack.org (or whatever url) at.  Then 
> > > > > > we
> will
> > > > have
> > > > > > an ASF
> > > > > >     > repo which tracks a mirror list and exposes a static 
> > > > > > site
> to
> > be
> > > > > able
> > > > > > to
> > > > > >     > query the mirror list.
> > > > > >     >
> > > > > >     > The web server will receive a request and will do a 
> > > > > > 302
> > > redirect
> > > > to
> > > > > > the
> > > > > >     > appropriate resource on one of the mirrors.
> > > > > >     >
> > > > > >     > This gets us started.  Legacy environments will have 
> > > > > > to do
> a
> > DB
> > > > > > change to
> > > > > >     > change from 'download.cloud.com' to '
> > downloads.cloudstack.org'
> > > > (or
> > > > > >     > whatever), but otherwise it should be pretty simple.
> > > > > >     >
> > > > > >     > We can improve how we deliver templates going forward, 
> > > > > > but
> > this
> > > > > > obviously
> > > > > >     > requires some discussion still.  I will try to buy us 
> > > > > > some
> > time
> > > > > with
> > > > > > an
> > > > > >     > implementation which solves our problems today...
> > > > > >     >
> > > > > >     > *Will STEVENS*
> > > > > >     > Lead Developer
> > > > > >     >
> > > > > >     > <https://goo.gl/NYZ8KK>
> > > > > >     >
> > > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > > > > >     > abhinandan.prateek@shapeblue.com> wrote:
> > > > > >     >
> > > > > >     >>
> > > > > >     >> Initial seeding is a manual step and that is not 
> > > > > > going to
> > > change
> > > > > in
> > > > > > near
> > > > > >     >> future. A handy list of official places from where 
> > > > > > these
> > > > templates
> > > > > > can be
> > > > > >     >> downloaded won’t harm this part of the installation. 
> > > > > > Just
> > > allow
> > > > > > anyone
> > > > > >     >> credible who follows democratic process to publish 
> > > > > > their
> > > > templates
> > > > > > with
> > > > > >     >> apache’s blessings.
> > > > > >     >>
> > > > > >     >>
> > > > > >     >> Coming to urls hard coded in db, the popular 
> > > > > > suggestion is
> > to
> > > > > point
> > > > > > them
> > > > > >     >> to mirrors. This looks good but this is going to take 
> > > > > > some
> > > > effort
> > > > > > that has
> > > > > >     >> not materialised yet. Once this change is made it 
> > > > > > will be
> > > going
> > > > in
> > > > > > some
> > > > > >     >> future release like 4.11; the problem still remains 
> > > > > > for
> the
> > > > > > releases that
> > > > > >     >> are already out there. In case access to
> download.cloud.com
> > > is
> > > > > > dropped
> > > > > >     >> at some point we will have to document the procedure 
> > > > > > such
> > that
> > > > > > people know
> > > > > >     >> how to make these stable releases work. That is where 
> > > > > > the
> > > > > procedure
> > > > > >     >> documented here https://shankerbalan.net/blog/
> > > > > >     >> seed-cloudstack-templates-offline/ should be 
> > > > > > officially
> > > adapted
> > > > > > with a
> > > > > >     >> marketplace to pick up templates of choosing. Maybe 
> > > > > > we can
> > > have
> > > > a
> > > > > > pre-setup
> > > > > >     >> script to make choices instead of written wiki. Note 
> > > > > > that
> > this
> > > > > will
> > > > > > be
> > > > > >     >> anyway needed by the older releases.
> > > > > >     >>
> > > > > >     >>
> > > > > >     >>
> > > > > >     >>
> > > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
> > paul.angus@shapeblue.com>
> > > > > > wrote:
> > > > > >     >>
> > > > > >     >> >-1
> > > > > >     >> >
> > > > > >     >> >We are trying to get MORE people to use CloudStack.  
> > > > > > And
> > > > raising
> > > > > > the
> > > > > >     >> barrier to entry does nothing to help that.
> > > > > >     >> >
> > > > > >     >> >
> > > > > >     >> >Kind regards,
> > > > > >     >> >
> > > > > >     >> >Paul Angus
> > > > > >     >> >
> > > > > >     >> >paul.angus@shapeblue.com
> > > > > >     >> >www.shapeblue.com
> > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > >     >> >@shapeblue
> > > > > >     >> >
> > > > > >     >> >
> > > > > >     >> >
> > > > > >     >> >
> > > > > >     >> >-----Original Message-----
> > > > > >     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@
> > > > > shapeblue.com]
> > > > > >     >> >Sent: 01 March 2017 09:39
> > > > > >     >> >To: dev@cloudstack.apache.org
> > > > > >     >> >Subject: Re: Modern template hosting
> > > > > >     >> >
> > > > > >     >> >+1
> > > > > >     >> >
> > > > > >     >> >After seeing this discussion running in circles 
> > > > > > several
> > > times,
> > > > I
> > > > > > think
> > > > > >     >> we should at least get started with the simplest option.
> > > > > >     >> >
> > > > > >     >> >
> > > > > >     >> >
> > > > > >     >> >
> > > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
> > > > chiradeepv@gmail.com>
> > > > > > wrote:
> > > > > >     >> >
> > > > > >     >> >>The alternative foolproof way is to simply not 
> > > > > > provide
> the
> > > > > > automatic
> > > > > >     >> >>download.
> > > > > >     >> >>
> > > > > >     >> >>Just document the mirror list URL where the 
> > > > > > templates
> can
> > be
> > > > > > found. I
> > > > > >     >> >>don't know why people reject this option.
> > > > > >     >> >>
> > > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
> > > > > templates-offline/
> > > > > >     >> >>
> > > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> > > > > >     >> >><wi...@gmail.com>
> > > > > >     >> >>wrote:
> > > > > >     >> >>
> > > > > >     >> >>> OK. Thanks for the heads up.
> > > > > >     >> >>>
> > > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" < 
> > > > > > chiradeepv@gmail.com>
> > > > > >     >> wrote:
> > > > > >     >> >>>
> > > > > >     >> >>> > Sounds workable. The downloader code in the 
> > > > > > SSVM
> won't
> > > > > follow
> > > > > >     >> >>> > redirects I think.
> > > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
> > > > > 5511065fc20787619d
> > > > > > 9cd0444
> > > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/
> storage/template/
> > > > > >     >> >>> > HttpTemplateDownloader.java#L93
> > > > > >     >> >>> > https://goo.gl/dSi0r5
> > > > > >     >> >>> >  Might need to add
> > > > > >     >> >>> > client.setRedirectStrategy(new
> LaxRedirectStrategy());
> > > > > >     >> >>> >
> > > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> > > > > >     >> >>> > <ws...@cloudops.com>
> > > > > >     >> >>> > wrote:
> > > > > >     >> >>> >
> > > > > >     >> >>> > > We haven't opened a ticket yet because we 
> > > > > > don't
> > have a
> > > > > > strategy
> > > > > >     >> yet.
> > > > > >     >> >>> > >
> > > > > >     >> >>> > > What do you guys think of this:
> > > > > >     >> >>> > > - We setup a new github repo in the 'apache' 
> > > > > > org
> > which
> > > > > > consists
> > > > > >     >> >>> > > of a
> > > > > >     >> >>> > single
> > > > > >     >> >>> > > file with a list of active/supported mirrors.
> > > > > >     >> >>> > > - I write a small web server, distributed as 
> > > > > > a
> > binary,
> > > > > > which can
> > > > > >     >> >>> > > be
> > > > > >     >> >>> > hosted
> > > > > >     >> >>> > > by ASF Infra.  This web server will query the
> > current
> > > > list
> > > > > > of
> > > > > >     >> >>> > > mirrors
> > > > > >     >> >>> and
> > > > > >     >> >>> > > will select one and then do a 302 redirect to 
> > > > > > that
> > > > mirror.
> > > > > >     >> >>> > >
> > > > > >     >> >>> > > The act of 'choosing' a mirror could be done 
> > > > > > in a
> > > number
> > > > > of
> > > > > > ways.
> > > > > >     >> >>> > > - If we want to define an order, then it 
> > > > > > could
> just
> > > try
> > > > > > from the
> > > > > >     >> >>> > > top of
> > > > > >     >> >>> > the
> > > > > >     >> >>> > > list and work its way down.  It would curl 
> > > > > > the
> > target
> > > to
> > > > > > make
> > > > > >     >> >>> > > sure it
> > > > > >     >> >>> > gets
> > > > > >     >> >>> > > a 200 and if it does, it would do a 302 redirect.
> > > > > >     >> >>> > > - Or, if we want to distribute the load 
> > > > > > across the
> > > > > mirrors,
> > > > > > we
> > > > > >     >> >>> > > could
> > > > > >     >> >>> pick
> > > > > >     >> >>> > > from the list randomly.  Again, doing a curl 
> > > > > > to
> > verify
> > > > the
> > > > > > mirror
> > > > > >     >> >>> > > is up
> > > > > >     >> >>> > and
> > > > > >     >> >>> > > then doing a redirect.
> > > > > >     >> >>> > > - If we want to get fancy, we could do a 
> > > > > > reverse
> IP
> > > > lookup
> > > > > > and
> > > > > >     >> >>> > > try to
> > > > > >     >> >>> > match
> > > > > >     >> >>> > > the requester with their closest geographical
> > mirror.
> > > > > >     >> >>> > >
> > > > > >     >> >>> > > Thoughts?
> > > > > >     >> >>> > >
> > > > > >     >> >>> > > *Will STEVENS*
> > > > > >     >> >>> > > Lead Developer
> > > > > >     >> >>> > >
> > > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
> > > > > >     >> >>> > >
> > > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep
> Vittal <
> > > > > >     >> >>> chiradeepv@gmail.com
> > > > > >     >> >>> > >
> > > > > >     >> >>> > > wrote:
> > > > > >     >> >>> > >
> > > > > >     >> >>> > > > What steps are needed to set up a mirror? 
> > > > > > What
> > does
> > > > > Infra
> > > > > > need
> > > > > >     >> to do?
> > > > > >     >> >>> > Has
> > > > > >     >> >>> > > > anybody filed a ticket with Infra?
> > > > > >     >> >>> > > >
> > > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > >     >> >>> > > > raja.pullela@accelerite.com>
> > > > > >     >> >>> > > > wrote:
> > > > > >     >> >>> > > >
> > > > > >     >> >>> > > > > Hi will,
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > I believe, we didn’t get to close 
> > > > > > ‘getting a
> > > mirror
> > > > on
> > > > > > Apache’
> > > > > >     >> >>> > because
> > > > > >     >> >>> > > we
> > > > > >     >> >>> > > > > needed someone on the Apache Infra side 
> > > > > > to
> close
> > > > this.
> > > > > > BTW,
> > > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux 
> > > > > > manages
> > this?)
> > > > has
> > > > > >     >> >>> > > > > all/most of
> > > > > >     >> >>> > the
> > > > > >     >> >>> > > > > content.  Once we can close on the Apache
> mirror
> > > for
> > > > > > hosting
> > > > > >     >> >>> > > > > the
> > > > > >     >> >>> > > > content, I
> > > > > >     >> >>> > > > > can help assist getting the content there.
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > For now, we have replicated the
> > > download.cloud.com
> > > > > > content to
> > > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
> > > > > >     >> >>> > > > > Also, we are working on a set of
> steps/procedure
> > > to
> > > > > > help with
> > > > > >     >> >>> > > > > this change.  I will update everyone in 
> > > > > > about
> a
> > > > week’s
> > > > > > time
> > > > > >     >> >>> > > > > on the
> > > > > >     >> >>> > details.
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > Best,
> > > > > >     >> >>> > > > > Raja Pullela
> > > > > >     >> >>> > > > > Engineering Team,
> > > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa 
> > > > > > Clara,
> > CA,
> > > > > 95054
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "
> williamstevens@gmail.com
> > > on
> > > > > > behalf of
> > > > > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com 
> > > > > > on
> > behalf
> > > > of
> > > > > >     >> >>> > wstevens@cloudops.com>
> > > > > >     >> >>> > > > > wrote:
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > unfortunately the template mirror 
> > > > > > conversation
> > got
> > > > > > caught up
> > > > > >     >> >>> > > > > in
> > > > > >     >> >>> > details
> > > > > >     >> >>> > > > and
> > > > > >     >> >>> > > > > nobody took the lead on implementing a
> solution.
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > citrix has been pinging me every couple 
> > > > > > months
> > to
> > > > say
> > > > > > 'dude,
> > > > > >     >> >>> > > > > we
> > > > > >     >> >>> need
> > > > > >     >> >>> > to
> > > > > >     >> >>> > > > > remove the dependency on 
> > > > > > download.citrix.com
> ',
> > > but
> > > > i
> > > > > > have not
> > > > > >     >> >>> > > > > had
> > > > > >     >> >>> > the
> > > > > >     >> >>> > > > > cycles to get in and solve the problem.  
> > > > > > the
> > > > shutdown
> > > > > > of that
> > > > > >     >> >>> > > > > is
> > > > > >     >> >>> > > imminent
> > > > > >     >> >>> > > > > right now, so we need to solve it asap.
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > *Will STEVENS*
> > > > > >     >> >>> > > > > Lead Developer
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > > >     >> >>> > paul.angus@shapeblue.com
> > > > > >     >> >>> > > >
> > > > > >     >> >>> > > > > wrote:
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > > Hi Nathan,
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > Ideally, if you put the template 
> > > > > > location in
> > (or
> > > > use
> > > > > a
> > > > > >     >> >>> > > > > > template
> > > > > >     >> >>> > > defined
> > > > > >     >> >>> > > > > > in)  test_data.py then the actual 
> > > > > > location
> can
> > > be
> > > > > >     >> >>> > > > > > overridden by
> > > > > >     >> >>> > > anyone
> > > > > >     >> >>> > > > > > testing.
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > For Trillian, we've copied all of the
> > templates
> > > > that
> > > > > > people
> > > > > >     >> >>> > > > > > have
> > > > > >     >> >>> > > define
> > > > > >     >> >>> > > > > to
> > > > > >     >> >>> > > > > > a local repo and then replace the URLs 
> > > > > > in test_data.py to
> > > > > >     >> >>> > > > > > reduce
> > > > > >     >> >>> > > > > bandwidth
> > > > > >     >> >>> > > > > > use and download times.
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > Ie:
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >             "bootableIso":
> > > > > >     >> >>> > > > > >                 {
> > > > > >     >> >>> > > > > >                     "displaytext": "Test
> > > Bootable
> > > > > > ISO",
> > > > > >     >> >>> > > > > >                     "name": "testISO",
> > > > > >     >> >>> > > > > >                     "bootable": True,
> > > > > >     >> >>> > > > > >                     "ispublic": False,
> > > > > >     >> >>> > > > > >                     "url": "{{
> > > > > marvin_images_location
> > > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
> > > > > >     >> >>> > > > > >                     "ostype": 'Other Linux
> > > > > (64-bit)',
> > > > > >     >> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > >     >> >>> > > > > >         },
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > I thought that we had come up with a
> solution
> > > for
> > > > > >     >> >>> > download.cloud.com
> > > > > >     >> >>> > > ,
> > > > > >     >> >>> > > > by
> > > > > >     >> >>> > > > > > having a mirrorlist hosted in Community
> Apache
> > > > > > 'space' with
> > > > > >     >> >>> anyone
> > > > > >     >> >>> > > able
> > > > > >     >> >>> > > > > to
> > > > > >     >> >>> > > > > > out themselves forward as a mirror.
> > > > > >     >> >>> > > > > > But I must admit I lost track of 
> > > > > > whether
> > anyone
> > > > made
> > > > > > the
> > > > > >     >> >>> requisite
> > > > > >     >> >>> > > > > changes
> > > > > >     >> >>> > > > > > in code....
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > Kind regards,
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > Paul Angus
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > paul.angus@shapeblue.com
> > > > > >     >> >>> > > > > > www.shapeblue.com
> > > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden, London
> WC2N
> > > > 4HSUK
> > > > > >     >> >>> > > > > > @shapeblue
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > -----Original Message-----
> > > > > >     >> >>> > > > > > From: williamstevens@gmail.com
> > > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
> > > > > >     >> >>> > On
> > > > > >     >> >>> > > > > > Behalf Of Will Stevens
> > > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
> > > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
> > > > > >     >> >>> > > > > > Subject: Re: Modern template hosting
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > this is a hard questions.  in general, 
> > > > > > we
> > should
> > > > be
> > > > > > setting
> > > > > >     >> >>> > > > > > up a
> > > > > >     >> >>> > > mirror
> > > > > >     >> >>> > > > > on
> > > > > >     >> >>> > > > > > some cloudstack/apache domain and then
> mirror
> > to
> > > > > other
> > > > > >     >> >>> > > > > > provided
> > > > > >     >> >>> > > > > templates.
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > we MUST come up with a solution to deprecate
> > > > > >     >> >>> > > > > > 'download.cloud.com
> > > > > >     >> >>> ',
> > > > > >     >> >>> > > > that
> > > > > >     >> >>> > > > > > is going to be going away any day now.
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > i don't know the right way to solve 
> > > > > > this to
> be
> > > > > > honest, but
> > > > > >     >> >>> > > > > > if you
> > > > > >     >> >>> > > have
> > > > > >     >> >>> > > > > > ideas, i am willing to help.
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > *Will STEVENS*
> > > > > >     >> >>> > > > > > Lead Developer
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, 
> > > > > > Nathan
> > > Johnson <
> > > > > >     >> >>> njohnson@ena.com
> > > > > >     >> >>> > >
> > > > > >     >> >>> > > > > wrote:
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > > So not to re-open a can of worms, but 
> > > > > > I’m
> > in a
> > > > > > situation
> > > > > >     >> >>> > > > > > > where
> > > > > >     >> >>> I
> > > > > >     >> >>> > > need
> > > > > >     >> >>> > > > > > > to come up with a Marvin component 
> > > > > > test
> that
> > > > > > depends on a
> > > > > >     >> >>> > template
> > > > > >     >> >>> > > > > > > based on a kernel that’s relatively 
> > > > > > new,
> > i.e.,
> > > > > > newer than
> > > > > >     >> >>> Centos
> > > > > >     >> >>> > > 5.3
> > > > > >     >> >>> > > > /
> > > > > >     >> >>> > > > > > Ubuntu 10.04 .
> > > > > >     >> >>> > > > > > > I see openvm.eu has a suitable 
> > > > > > template
> > > (Ubuntu
> > > > > > 16.0.4
> > > > > >     >> >>> > > > > > > for
> > > > > >     >> >>> KVM),
> > > > > >     >> >>> > > but
> > > > > >     >> >>> > > > > > > from looking at the thread "Migrating
> > > CloudStack
> > > > > > content
> > > > > >     >> >>> > > > > > > from download.cloud.com” it looks 
> > > > > > like
> > there
> > > is
> > > > > >     >> >>> > > > > > > resistance to using
> > > > > >     >> >>> > > this
> > > > > >     >> >>> > > > at
> > > > > >     >> >>> > > > > > > least for hosting system vm templates 
> > > > > > over
> > > > > concerns
> > > > > > of
> > > > > >     >> >>> > neutrality.
> > > > > >     >> >>> > > > > > > Would this be suitable for a 
> > > > > > component
> test?
> > > If
> > > > > > not,
> > > > > >     >> >>> > > > > > > what is a
> > > > > >     >> >>> > > > > “blessed”
> > > > > >     >> >>> > > > > > > template location?
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > > Thanks in advance!
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > > Nathan Johnson
> > > > > >     >> >>> > > > > > > R&D Engineer
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12
> > Nashville,
> > > TN
> > > > > > 37211
> > > > > >     >> >>> > > > > > > General Office: 615-312-6000
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > > website | blog | support
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > DISCLAIMER
> > > > > >     >> >>> > > > > ==========
> > > > > >     >> >>> > > > > This e-mail may contain privileged and
> > > confidential
> > > > > >     >> >>> > > > > information
> > > > > >     >> >>> which
> > > > > >     >> >>> > > is
> > > > > >     >> >>> > > > > the property of Accelerite, a Persistent
> Systems
> > > > > > business. It
> > > > > >     >> >>> > > > > is
> > > > > >     >> >>> > > intended
> > > > > >     >> >>> > > > > only for the use of the individual or 
> > > > > > entity
> to
> > > > which
> > > > > > it is
> > > > > >     >> >>> > addressed.
> > > > > >     >> >>> > > If
> > > > > >     >> >>> > > > > you are not the intended recipient, you 
> > > > > > are
> not
> > > > > > authorized to
> > > > > >     >> >>> > > > > read,
> > > > > >     >> >>> > > > retain,
> > > > > >     >> >>> > > > > copy, print, distribute or use this message.
> If
> > > you
> > > > > have
> > > > > >     >> >>> > > > > received
> > > > > >     >> >>> > this
> > > > > >     >> >>> > > > > communication in error, please notify the
> sender
> > > and
> > > > > > delete
> > > > > >     >> >>> > > > > all
> > > > > >     >> >>> > copies
> > > > > >     >> >>> > > of
> > > > > >     >> >>> > > > > this message. Accelerite, a Persistent 
> > > > > > Systems
> > > > > business
> > > > > > does
> > > > > >     >> >>> > > > > not
> > > > > >     >> >>> > accept
> > > > > >     >> >>> > > > any
> > > > > >     >> >>> > > > > liability for virus infected mails.
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > >
> > > > > >     >> >>> > >
> > > > > >     >> >>> >
> > > > > >     >> >>>
> > > > > >     >> >
> > > > > >     >> >abhinandan.prateek@shapeblue.com
> > > > > >     >> >www.shapeblue.com
> > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > @shapeblue
> > > > > >     >> >
> > > > > >     >> >
> > > > > >     >> >
> > > > > >     >>
> > > > > >     >> abhinandan.prateek@shapeblue.com
> > > > > >     >> www.shapeblue.com
> > > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > >     >> @shapeblue
> > > > > >     >>
> > > > > >     >>
> > > > > >     >>
> > > > > >     >>
> > > > > >     >
> > > > > >
> > > > > >
> > > > > >
> > > > > > daan.hoogland@shapeblue.com
> > > > > > www.shapeblue.com
> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK 
> > > > > > @shapeblue
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rafael Weingärtner
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Rafael Weingärtner
> > > >
> > >
> >
>

Re: Modern template hosting

Posted by Chiradeep Vittal <ch...@gmail.com>.

I do feel like this is early optimization. Mirrors rarely fail. I'd expect
a single web server hosted on Apache Infra without any monitors to fail
more often than a mirror. We already expect Wido's systemvm repository to
be up all the time. And it has been. Similarly, I don't believe Nux's
repository has ever been down. And if Accelerite wants to host on S3, that
one is pretty solid as well.

This is an infrequent operation in a cloud. After the cloud is installed,
the download servers are only needed for a new zone. If we trust the user
to run a cloud, surely he/she can run a web server to serve some built-in
templates. And if her cloud is successful, she needs to figure out how to
host her templates anyway and not rely on 3rd parties.


On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <ws...@cloudops.com> wrote:

> 1) If the legacy implementations do not support redirects, that does cause
> a problem.  A potential solution in that case is to have the web server
> actually proxy the download, but that is not ideal and I would like to
> avoid it if possible.  Thanks for bringing that up Chiradeep.
>
> 2) I think we need to have a single URL which people can target.  Once they
> make the switch to the new URL, we want the implementation to be able to
> handle mirror failures without affecting the end client.  We want to avoid
> the situation where an ACS user will ever have to change this URL more than
> once.  Mirror failures SHOULD NOT affect the ACS users assuming there is
> still at least one mirror who can serve the requested resource.
>
> These are obviously my personal opinions and others will probably have
> differing opinions.
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal <ch...@gmail.com>
> wrote:
>
> > 1. If you are targeting legacy installations, they are not able to follow
> > redirects. The line of code that added this capability was added on
> 11/16.
> > 2. If you trust the users to edit the database to change the URL, you can
> > trust them to change it to anything. Just document a known list of good
> > template locations.
> >
> > On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <ws...@cloudops.com>
> > wrote:
> >
> > > So the main issue I see with this is the following.
> > >
> > > This implementation is designed to target legacy installations which
> will
> > > be affected when download.cloud.com disappears.  These people will
> need
> > to
> > > make a DB change to replace the 'download.cloud.com' with some other
> url
> > > (maybe 'download.cloudstack.org').  Once the DB has been updated, we
> can
> > > not expect anything else of the client, they need to be able to just
> > > continue operation as they were without any need to rebuild or upgrade
> > > their ACS.
> > >
> > > If we try to force the decision to the client, then we break the legacy
> > > implementations.  Since the SSVM is likely going to be the client in
> some
> > > cases, and since it does not already have the logic to handle the 300
> > > approach correctly, I feel like it is not a viable solution for the
> > legacy
> > > deployments.
> > >
> > > In addition to that.  I don't think the client is in any better
> position
> > to
> > > make the mirror decision than I am.  I am likely able to make a more
> > > 'educated' decision than the client would be able to because I can do
> > > different tests on the endpoint(s) before making a final decision.
> > >
> > > Does this make sense or am I missing something obvious?
> > >
> > > *Will STEVENS*
> > > Lead Developer
> > >
> > > <https://goo.gl/NYZ8KK>
> > >
> > > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
> > > rafaelweingartner@gmail.com> wrote:
> > >
> > > > I think I understood you, but I did not understand you.
> > > >
> > > > Let me see if I can create an example to illustrate how we could do
> > using
> > > > the HTTP 300 code.
> > > >
> > > > The HTTP 300 code indicates clients that he/she/it has multiple
> choices
> > > to
> > > > access the requested resource. If the request is a HEAD, this is the
> > only
> > > > thing that the server returns. If the request if a GET, then the
> server
> > > > returns the choices for the client. The response type is defined on
> the
> > > > server based on the Content-Type, user agent and others.
> > > >
> > > > Let’s say the server receives a GET request and informs the
> > content-type
> > > as
> > > > JSON (application/JSON). Your application could get the mirrors list,
> > do
> > > > the appending process for the requested resource path, and then
> return
> > a
> > > > JSON array with the possible mirror the client can use. Then, the
> > client
> > > > gets this list and does the processing required to select a mirror.
> > > >
> > > > Now let’s say that a human access the link using a browser. Then, the
> > > > server should receive something like (text/plain) as the
> content-type;
> > I
> > > > think depending on the browser this field may differ a little (it
> would
> > > > require some checking). Anyways, the server detects that it is a
> > “human”
> > > > requesting the resource, then we could serve an HTML page with a
> > > Javascript
> > > > that uses the mirror list. This Javascript could do some checking and
> > > > choose the best mirror for that specific.
> > > >
> > > > Why do I think this approach is interesting?
> > > > The application you developed would be used only to retrieve valid
> and
> > > > trusted sources of system VM images (a trusted repository for mirrors
> > > that
> > > > we as a community vouch for).  Also, we shift the decision process
> > > > regarding mirrors from the server to the client. Then, it is up to
> > > clients
> > > > to select mirrors, and not up to us (or some of our implementations).
> > > >
> > > > Did this help?
> > > > BTW: I am not saying I am against the way you proposed, which would
> > work
> > > > fine. It is merely a suggestion using a different perspective.
> > > >
> > > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
> > williamstevens@gmail.com>
> > > > wrote:
> > > >
> > > > > @rafael: in general, I don't think the client should ever be
> making a
> > > > > choice about a mirror. We have to assume we are working with a
> > scripted
> > > > > application and anything hard coded in that implementation on the
> > > client
> > > > > side is a risk.
> > > > >
> > > > > I may not be understanding the use of the 300 approach, so I will
> > > > research
> > > > > it to see if I can make it fit.
> > > > >
> > > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
> > > > rafaelweingartner@gmail.com>
> > > > > wrote:
> > > > >
> > > > > Will great job.
> > > > >
> > > > > I had the same doubt as Daan.
> > > > >
> > > > > Have you considered using HTTP 300 working mode? Then, we could let
> > the
> > > > > client decide which mirror is the best(closest?). Section “10.3.1”
> of
> > > [1]
> > > > > talks about it; this would put the pressure on deciding upon a
> mirror
> > > on
> > > > > the client side, but I think it would be fair.
> > > > >
> > > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
> > > > >
> > > > > PS: You really like very short variables names!
> > > > >
> > > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
> > > > daan.hoogland@shapeblue.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Nice little thing Will,
> > > > > >
> > > > > > One question: if I read the code correctly it ‘go’es and tries
> all
> > > > > mirrors
> > > > > > at once and whichever responses first is redirected to the
> client.
> > > This
> > > > > > might well be the same every time. This might be the one closest
> to
> > > the
> > > > > > mirror302 and not to the client. Is that correct or did I miss a
> > > > weighing
> > > > > > algorithm hidden in there?
> > > > > >
> > > > > > Good coding,
> > > > > >
> > > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
> > > > > >
> > > > > >     Hey All,
> > > > > >     Please review this repo and tell me what you think:
> > > > > >     https://github.com/swill/mirror302
> > > > > >
> > > > > >     Let me know if you have questions or you would like me to
> make
> > > > > changes
> > > > > >     before I bring this topic up with ASF Infra and open a ticket
> > to
> > > > get
> > > > > > this
> > > > > >     implemented.
> > > > > >
> > > > > >     Cheers,
> > > > > >
> > > > > >     *Will Stevens*
> > > > > >
> > > > > >
> > > > > >
> > > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> > > > wstevens@cloudops.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >     > I am building a short term solution right now.  I hopefully
> > > will
> > > > > > have the
> > > > > >     > building blocks in place by the weekend so we can start
> > working
> > > > > with
> > > > > > Infra
> > > > > >     > to get it in place.
> > > > > >     >
> > > > > >     > We will have a web server which we can point something like
> > > > > >     > downloads.cloudstack.org (or whatever url) at.  Then we
> will
> > > > have
> > > > > > an ASF
> > > > > >     > repo which tracks a mirror list and exposes a static site
> to
> > be
> > > > > able
> > > > > > to
> > > > > >     > query the mirror list.
> > > > > >     >
> > > > > >     > The web server will receive a request and will do a 302
> > > redirect
> > > > to
> > > > > > the
> > > > > >     > appropriate resource on one of the mirrors.
> > > > > >     >
> > > > > >     > This gets us started.  Legacy environments will have to do
> a
> > DB
> > > > > > change to
> > > > > >     > change from 'download.cloud.com' to '
> > downloads.cloudstack.org'
> > > > (or
> > > > > >     > whatever), but otherwise it should be pretty simple.
> > > > > >     >
> > > > > >     > We can improve how we deliver templates going forward, but
> > this
> > > > > > obviously
> > > > > >     > requires some discussion still.  I will try to buy us some
> > time
> > > > > with
> > > > > > an
> > > > > >     > implementation which solves our problems today...
> > > > > >     >
> > > > > >     > *Will STEVENS*
> > > > > >     > Lead Developer
> > > > > >     >
> > > > > >     > <https://goo.gl/NYZ8KK>
> > > > > >     >
> > > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > > > > >     > abhinandan.prateek@shapeblue.com> wrote:
> > > > > >     >
> > > > > >     >>
> > > > > >     >> Initial seeding is a manual step and that is not going to
> > > change
> > > > > in
> > > > > > near
> > > > > >     >> future. A handy list of official places from where these
> > > > templates
> > > > > > can be
> > > > > >     >> downloaded won’t harm this part of the installation. Just
> > > allow
> > > > > > anyone
> > > > > >     >> credible who follows democratic process to publish their
> > > > templates
> > > > > > with
> > > > > >     >> apache’s blessings.
> > > > > >     >>
> > > > > >     >>
> > > > > >     >> Coming to urls hard coded in db, the popular suggestion is
> > to
> > > > > point
> > > > > > them
> > > > > >     >> to mirrors. This looks good but this is going to take some
> > > > effort
> > > > > > that has
> > > > > >     >> not materialised yet. Once this change is made it will be
> > > going
> > > > in
> > > > > > some
> > > > > >     >> future release like 4.11; the problem still remains for
> the
> > > > > > releases that
> > > > > >     >> are already out there. In case access to
> download.cloud.com
> > > is
> > > > > > dropped
> > > > > >     >> at some point we will have to document the procedure such
> > that
> > > > > > people know
> > > > > >     >> how to make these stable releases work. That is where the
> > > > > procedure
> > > > > >     >> documented here https://shankerbalan.net/blog/
> > > > > >     >> seed-cloudstack-templates-offline/ should be officially
> > > adapted
> > > > > > with a
> > > > > >     >> marketplace to pick up templates of choosing. Maybe we can
> > > have
> > > > a
> > > > > > pre-setup
> > > > > >     >> script to make choices instead of written wiki. Note that
> > this
> > > > > will
> > > > > > be
> > > > > >     >> anyway needed by the older releases.
> > > > > >     >>
> > > > > >     >>
> > > > > >     >>
> > > > > >     >>
> > > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
> > paul.angus@shapeblue.com>
> > > > > > wrote:
> > > > > >     >>
> > > > > >     >> >-1
> > > > > >     >> >
> > > > > >     >> >We are trying to get MORE people to use CloudStack.  And
> > > > raising
> > > > > > the
> > > > > >     >> barrier to entry does nothing to help that.
> > > > > >     >> >
> > > > > >     >> >
> > > > > >     >> >Kind regards,
> > > > > >     >> >
> > > > > >     >> >Paul Angus
> > > > > >     >> >
> > > > > >     >> >paul.angus@shapeblue.com
> > > > > >     >> >www.shapeblue.com
> > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > >     >> >@shapeblue
> > > > > >     >> >
> > > > > >     >> >
> > > > > >     >> >
> > > > > >     >> >
> > > > > >     >> >-----Original Message-----
> > > > > >     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@
> > > > > shapeblue.com]
> > > > > >     >> >Sent: 01 March 2017 09:39
> > > > > >     >> >To: dev@cloudstack.apache.org
> > > > > >     >> >Subject: Re: Modern template hosting
> > > > > >     >> >
> > > > > >     >> >+1
> > > > > >     >> >
> > > > > >     >> >After seeing this discussion running in circles several
> > > times,
> > > > I
> > > > > > think
> > > > > >     >> we should at least get started with the simplest option.
> > > > > >     >> >
> > > > > >     >> >
> > > > > >     >> >
> > > > > >     >> >
> > > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
> > > > chiradeepv@gmail.com>
> > > > > > wrote:
> > > > > >     >> >
> > > > > >     >> >>The alternative foolproof way is to simply not provide
> the
> > > > > > automatic
> > > > > >     >> >>download.
> > > > > >     >> >>
> > > > > >     >> >>Just document the mirror list URL where the templates
> can
> > be
> > > > > > found. I
> > > > > >     >> >>don't know why people reject this option.
> > > > > >     >> >>
> > > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
> > > > > templates-offline/
> > > > > >     >> >>
> > > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> > > > > >     >> >><wi...@gmail.com>
> > > > > >     >> >>wrote:
> > > > > >     >> >>
> > > > > >     >> >>> OK. Thanks for the heads up.
> > > > > >     >> >>>
> > > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > > > chiradeepv@gmail.com>
> > > > > >     >> wrote:
> > > > > >     >> >>>
> > > > > >     >> >>> > Sounds workable. The downloader code in the SSVM
> won't
> > > > > follow
> > > > > >     >> >>> > redirects I think.
> > > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
> > > > > 5511065fc20787619d
> > > > > > 9cd0444
> > > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/
> storage/template/
> > > > > >     >> >>> > HttpTemplateDownloader.java#L93
> > > > > >     >> >>> > https://goo.gl/dSi0r5
> > > > > >     >> >>> >  Might need to add
> > > > > >     >> >>> > client.setRedirectStrategy(new
> LaxRedirectStrategy());
> > > > > >     >> >>> >
> > > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> > > > > >     >> >>> > <ws...@cloudops.com>
> > > > > >     >> >>> > wrote:
> > > > > >     >> >>> >
> > > > > >     >> >>> > > We haven't opened a ticket yet because we don't
> > have a
> > > > > > strategy
> > > > > >     >> yet.
> > > > > >     >> >>> > >
> > > > > >     >> >>> > > What do you guys think of this:
> > > > > >     >> >>> > > - We setup a new github repo in the 'apache' org
> > which
> > > > > > consists
> > > > > >     >> >>> > > of a
> > > > > >     >> >>> > single
> > > > > >     >> >>> > > file with a list of active/supported mirrors.
> > > > > >     >> >>> > > - I write a small web server, distributed as a
> > binary,
> > > > > > which can
> > > > > >     >> >>> > > be
> > > > > >     >> >>> > hosted
> > > > > >     >> >>> > > by ASF Infra.  This web server will query the
> > current
> > > > list
> > > > > > of
> > > > > >     >> >>> > > mirrors
> > > > > >     >> >>> and
> > > > > >     >> >>> > > will select one and then do a 302 redirect to that
> > > > mirror.
> > > > > >     >> >>> > >
> > > > > >     >> >>> > > The act of 'choosing' a mirror could be done in a
> > > number
> > > > > of
> > > > > > ways.
> > > > > >     >> >>> > > - If we want to define an order, then it could
> just
> > > try
> > > > > > from the
> > > > > >     >> >>> > > top of
> > > > > >     >> >>> > the
> > > > > >     >> >>> > > list and work its way down.  It would curl the
> > target
> > > to
> > > > > > make
> > > > > >     >> >>> > > sure it
> > > > > >     >> >>> > gets
> > > > > >     >> >>> > > a 200 and if it does, it would do a 302 redirect.
> > > > > >     >> >>> > > - Or, if we want to distribute the load across the
> > > > > mirrors,
> > > > > > we
> > > > > >     >> >>> > > could
> > > > > >     >> >>> pick
> > > > > >     >> >>> > > from the list randomly.  Again, doing a curl to
> > verify
> > > > the
> > > > > > mirror
> > > > > >     >> >>> > > is up
> > > > > >     >> >>> > and
> > > > > >     >> >>> > > then doing a redirect.
> > > > > >     >> >>> > > - If we want to get fancy, we could do a reverse
> IP
> > > > lookup
> > > > > > and
> > > > > >     >> >>> > > try to
> > > > > >     >> >>> > match
> > > > > >     >> >>> > > the requester with their closest geographical
> > mirror.
> > > > > >     >> >>> > >
> > > > > >     >> >>> > > Thoughts?
> > > > > >     >> >>> > >
> > > > > >     >> >>> > > *Will STEVENS*
> > > > > >     >> >>> > > Lead Developer
> > > > > >     >> >>> > >
> > > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
> > > > > >     >> >>> > >
> > > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep
> Vittal <
> > > > > >     >> >>> chiradeepv@gmail.com
> > > > > >     >> >>> > >
> > > > > >     >> >>> > > wrote:
> > > > > >     >> >>> > >
> > > > > >     >> >>> > > > What steps are needed to set up a mirror? What
> > does
> > > > > Infra
> > > > > > need
> > > > > >     >> to do?
> > > > > >     >> >>> > Has
> > > > > >     >> >>> > > > anybody filed a ticket with Infra?
> > > > > >     >> >>> > > >
> > > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > >     >> >>> > > > raja.pullela@accelerite.com>
> > > > > >     >> >>> > > > wrote:
> > > > > >     >> >>> > > >
> > > > > >     >> >>> > > > > Hi will,
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > I believe, we didn’t get to close ‘getting a
> > > mirror
> > > > on
> > > > > > Apache’
> > > > > >     >> >>> > because
> > > > > >     >> >>> > > we
> > > > > >     >> >>> > > > > needed someone on the Apache Infra side to
> close
> > > > this.
> > > > > > BTW,
> > > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages
> > this?)
> > > > has
> > > > > >     >> >>> > > > > all/most of
> > > > > >     >> >>> > the
> > > > > >     >> >>> > > > > content.  Once we can close on the Apache
> mirror
> > > for
> > > > > > hosting
> > > > > >     >> >>> > > > > the
> > > > > >     >> >>> > > > content, I
> > > > > >     >> >>> > > > > can help assist getting the content there.
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > For now, we have replicated the
> > > download.cloud.com
> > > > > > content to
> > > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
> > > > > >     >> >>> > > > > Also, we are working on a set of
> steps/procedure
> > > to
> > > > > > help with
> > > > > >     >> >>> > > > > this change.  I will update everyone in about
> a
> > > > week’s
> > > > > > time
> > > > > >     >> >>> > > > > on the
> > > > > >     >> >>> > details.
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > Best,
> > > > > >     >> >>> > > > > Raja Pullela
> > > > > >     >> >>> > > > > Engineering Team,
> > > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara,
> > CA,
> > > > > 95054
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "
> williamstevens@gmail.com
> > > on
> > > > > > behalf of
> > > > > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com on
> > behalf
> > > > of
> > > > > >     >> >>> > wstevens@cloudops.com>
> > > > > >     >> >>> > > > > wrote:
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > unfortunately the template mirror conversation
> > got
> > > > > > caught up
> > > > > >     >> >>> > > > > in
> > > > > >     >> >>> > details
> > > > > >     >> >>> > > > and
> > > > > >     >> >>> > > > > nobody took the lead on implementing a
> solution.
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > citrix has been pinging me every couple months
> > to
> > > > say
> > > > > > 'dude,
> > > > > >     >> >>> > > > > we
> > > > > >     >> >>> need
> > > > > >     >> >>> > to
> > > > > >     >> >>> > > > > remove the dependency on download.citrix.com
> ',
> > > but
> > > > i
> > > > > > have not
> > > > > >     >> >>> > > > > had
> > > > > >     >> >>> > the
> > > > > >     >> >>> > > > > cycles to get in and solve the problem.  the
> > > > shutdown
> > > > > > of that
> > > > > >     >> >>> > > > > is
> > > > > >     >> >>> > > imminent
> > > > > >     >> >>> > > > > right now, so we need to solve it asap.
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > *Will STEVENS*
> > > > > >     >> >>> > > > > Lead Developer
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > > >     >> >>> > paul.angus@shapeblue.com
> > > > > >     >> >>> > > >
> > > > > >     >> >>> > > > > wrote:
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > > Hi Nathan,
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > Ideally, if you put the template location in
> > (or
> > > > use
> > > > > a
> > > > > >     >> >>> > > > > > template
> > > > > >     >> >>> > > defined
> > > > > >     >> >>> > > > > > in)  test_data.py then the actual location
> can
> > > be
> > > > > >     >> >>> > > > > > overridden by
> > > > > >     >> >>> > > anyone
> > > > > >     >> >>> > > > > > testing.
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > For Trillian, we've copied all of the
> > templates
> > > > that
> > > > > > people
> > > > > >     >> >>> > > > > > have
> > > > > >     >> >>> > > define
> > > > > >     >> >>> > > > > to
> > > > > >     >> >>> > > > > > a local repo and then replace the URLs in
> > > > > > test_data.py to
> > > > > >     >> >>> > > > > > reduce
> > > > > >     >> >>> > > > > bandwidth
> > > > > >     >> >>> > > > > > use and download times.
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > Ie:
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >             "bootableIso":
> > > > > >     >> >>> > > > > >                 {
> > > > > >     >> >>> > > > > >                     "displaytext": "Test
> > > Bootable
> > > > > > ISO",
> > > > > >     >> >>> > > > > >                     "name": "testISO",
> > > > > >     >> >>> > > > > >                     "bootable": True,
> > > > > >     >> >>> > > > > >                     "ispublic": False,
> > > > > >     >> >>> > > > > >                     "url": "{{
> > > > > marvin_images_location
> > > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
> > > > > >     >> >>> > > > > >                     "ostype": 'Other Linux
> > > > > (64-bit)',
> > > > > >     >> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > >     >> >>> > > > > >         },
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > I thought that we had come up with a
> solution
> > > for
> > > > > >     >> >>> > download.cloud.com
> > > > > >     >> >>> > > ,
> > > > > >     >> >>> > > > by
> > > > > >     >> >>> > > > > > having a mirrorlist hosted in Community
> Apache
> > > > > > 'space' with
> > > > > >     >> >>> anyone
> > > > > >     >> >>> > > able
> > > > > >     >> >>> > > > > to
> > > > > >     >> >>> > > > > > out themselves forward as a mirror.
> > > > > >     >> >>> > > > > > But I must admit I lost track of whether
> > anyone
> > > > made
> > > > > > the
> > > > > >     >> >>> requisite
> > > > > >     >> >>> > > > > changes
> > > > > >     >> >>> > > > > > in code....
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > Kind regards,
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > Paul Angus
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > paul.angus@shapeblue.com
> > > > > >     >> >>> > > > > > www.shapeblue.com
> > > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden, London
> WC2N
> > > > 4HSUK
> > > > > >     >> >>> > > > > > @shapeblue
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > -----Original Message-----
> > > > > >     >> >>> > > > > > From: williamstevens@gmail.com
> > > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
> > > > > >     >> >>> > On
> > > > > >     >> >>> > > > > > Behalf Of Will Stevens
> > > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
> > > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
> > > > > >     >> >>> > > > > > Subject: Re: Modern template hosting
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > this is a hard questions.  in general, we
> > should
> > > > be
> > > > > > setting
> > > > > >     >> >>> > > > > > up a
> > > > > >     >> >>> > > mirror
> > > > > >     >> >>> > > > > on
> > > > > >     >> >>> > > > > > some cloudstack/apache domain and then
> mirror
> > to
> > > > > other
> > > > > >     >> >>> > > > > > provided
> > > > > >     >> >>> > > > > templates.
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > we MUST come up with a solution to deprecate
> > > > > >     >> >>> > > > > > 'download.cloud.com
> > > > > >     >> >>> ',
> > > > > >     >> >>> > > > that
> > > > > >     >> >>> > > > > > is going to be going away any day now.
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > i don't know the right way to solve this to
> be
> > > > > > honest, but
> > > > > >     >> >>> > > > > > if you
> > > > > >     >> >>> > > have
> > > > > >     >> >>> > > > > > ideas, i am willing to help.
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > *Will STEVENS*
> > > > > >     >> >>> > > > > > Lead Developer
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan
> > > Johnson <
> > > > > >     >> >>> njohnson@ena.com
> > > > > >     >> >>> > >
> > > > > >     >> >>> > > > > wrote:
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > > > > So not to re-open a can of worms, but I’m
> > in a
> > > > > > situation
> > > > > >     >> >>> > > > > > > where
> > > > > >     >> >>> I
> > > > > >     >> >>> > > need
> > > > > >     >> >>> > > > > > > to come up with a Marvin component test
> that
> > > > > > depends on a
> > > > > >     >> >>> > template
> > > > > >     >> >>> > > > > > > based on a kernel that’s relatively new,
> > i.e.,
> > > > > > newer than
> > > > > >     >> >>> Centos
> > > > > >     >> >>> > > 5.3
> > > > > >     >> >>> > > > /
> > > > > >     >> >>> > > > > > Ubuntu 10.04 .
> > > > > >     >> >>> > > > > > > I see openvm.eu has a suitable template
> > > (Ubuntu
> > > > > > 16.0.4
> > > > > >     >> >>> > > > > > > for
> > > > > >     >> >>> KVM),
> > > > > >     >> >>> > > but
> > > > > >     >> >>> > > > > > > from looking at the thread "Migrating
> > > CloudStack
> > > > > > content
> > > > > >     >> >>> > > > > > > from download.cloud.com” it looks like
> > there
> > > is
> > > > > >     >> >>> > > > > > > resistance to using
> > > > > >     >> >>> > > this
> > > > > >     >> >>> > > > at
> > > > > >     >> >>> > > > > > > least for hosting system vm templates over
> > > > > concerns
> > > > > > of
> > > > > >     >> >>> > neutrality.
> > > > > >     >> >>> > > > > > > Would this be suitable for a component
> test?
> > > If
> > > > > > not,
> > > > > >     >> >>> > > > > > > what is a
> > > > > >     >> >>> > > > > “blessed”
> > > > > >     >> >>> > > > > > > template location?
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > > Thanks in advance!
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > > Nathan Johnson
> > > > > >     >> >>> > > > > > > R&D Engineer
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12
> > Nashville,
> > > TN
> > > > > > 37211
> > > > > >     >> >>> > > > > > > General Office: 615-312-6000
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > > website | blog | support
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > > >
> > > > > >     >> >>> > > > > >
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > > > DISCLAIMER
> > > > > >     >> >>> > > > > ==========
> > > > > >     >> >>> > > > > This e-mail may contain privileged and
> > > confidential
> > > > > >     >> >>> > > > > information
> > > > > >     >> >>> which
> > > > > >     >> >>> > > is
> > > > > >     >> >>> > > > > the property of Accelerite, a Persistent
> Systems
> > > > > > business. It
> > > > > >     >> >>> > > > > is
> > > > > >     >> >>> > > intended
> > > > > >     >> >>> > > > > only for the use of the individual or entity
> to
> > > > which
> > > > > > it is
> > > > > >     >> >>> > addressed.
> > > > > >     >> >>> > > If
> > > > > >     >> >>> > > > > you are not the intended recipient, you are
> not
> > > > > > authorized to
> > > > > >     >> >>> > > > > read,
> > > > > >     >> >>> > > > retain,
> > > > > >     >> >>> > > > > copy, print, distribute or use this message.
> If
> > > you
> > > > > have
> > > > > >     >> >>> > > > > received
> > > > > >     >> >>> > this
> > > > > >     >> >>> > > > > communication in error, please notify the
> sender
> > > and
> > > > > > delete
> > > > > >     >> >>> > > > > all
> > > > > >     >> >>> > copies
> > > > > >     >> >>> > > of
> > > > > >     >> >>> > > > > this message. Accelerite, a Persistent Systems
> > > > > business
> > > > > > does
> > > > > >     >> >>> > > > > not
> > > > > >     >> >>> > accept
> > > > > >     >> >>> > > > any
> > > > > >     >> >>> > > > > liability for virus infected mails.
> > > > > >     >> >>> > > > >
> > > > > >     >> >>> > > >
> > > > > >     >> >>> > >
> > > > > >     >> >>> >
> > > > > >     >> >>>
> > > > > >     >> >
> > > > > >     >> >abhinandan.prateek@shapeblue.com
> > > > > >     >> >www.shapeblue.com
> > > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > @shapeblue
> > > > > >     >> >
> > > > > >     >> >
> > > > > >     >> >
> > > > > >     >>
> > > > > >     >> abhinandan.prateek@shapeblue.com
> > > > > >     >> www.shapeblue.com
> > > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > >     >> @shapeblue
> > > > > >     >>
> > > > > >     >>
> > > > > >     >>
> > > > > >     >>
> > > > > >     >
> > > > > >
> > > > > >
> > > > > >
> > > > > > daan.hoogland@shapeblue.com
> > > > > > www.shapeblue.com
> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > @shapeblue
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rafael Weingärtner
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Rafael Weingärtner
> > > >
> > >
> >
>

Re: Modern template hosting

Posted by Will Stevens <ws...@cloudops.com>.

1) If the legacy implementations do not support redirects, that does cause
a problem.  A potential solution in that case is to have the web server
actually proxy the download, but that is not ideal and I would like to
avoid it if possible.  Thanks for bringing that up Chiradeep.

2) I think we need to have a single URL which people can target.  Once they
make the switch to the new URL, we want the implementation to be able to
handle mirror failures without affecting the end client.  We want to avoid
the situation where an ACS user will ever have to change this URL more than
once.  Mirror failures SHOULD NOT affect the ACS users assuming there is
still at least one mirror who can serve the requested resource.

These are obviously my personal opinions and others will probably have
differing opinions.

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal <ch...@gmail.com>
wrote:

> 1. If you are targeting legacy installations, they are not able to follow
> redirects. The line of code that added this capability was added on 11/16.
> 2. If you trust the users to edit the database to change the URL, you can
> trust them to change it to anything. Just document a known list of good
> template locations.
>
> On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <ws...@cloudops.com>
> wrote:
>
> > So the main issue I see with this is the following.
> >
> > This implementation is designed to target legacy installations which will
> > be affected when download.cloud.com disappears.  These people will need
> to
> > make a DB change to replace the 'download.cloud.com' with some other url
> > (maybe 'download.cloudstack.org').  Once the DB has been updated, we can
> > not expect anything else of the client, they need to be able to just
> > continue operation as they were without any need to rebuild or upgrade
> > their ACS.
> >
> > If we try to force the decision to the client, then we break the legacy
> > implementations.  Since the SSVM is likely going to be the client in some
> > cases, and since it does not already have the logic to handle the 300
> > approach correctly, I feel like it is not a viable solution for the
> legacy
> > deployments.
> >
> > In addition to that.  I don't think the client is in any better position
> to
> > make the mirror decision than I am.  I am likely able to make a more
> > 'educated' decision than the client would be able to because I can do
> > different tests on the endpoint(s) before making a final decision.
> >
> > Does this make sense or am I missing something obvious?
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
> > rafaelweingartner@gmail.com> wrote:
> >
> > > I think I understood you, but I did not understand you.
> > >
> > > Let me see if I can create an example to illustrate how we could do
> using
> > > the HTTP 300 code.
> > >
> > > The HTTP 300 code indicates clients that he/she/it has multiple choices
> > to
> > > access the requested resource. If the request is a HEAD, this is the
> only
> > > thing that the server returns. If the request if a GET, then the server
> > > returns the choices for the client. The response type is defined on the
> > > server based on the Content-Type, user agent and others.
> > >
> > > Let’s say the server receives a GET request and informs the
> content-type
> > as
> > > JSON (application/JSON). Your application could get the mirrors list,
> do
> > > the appending process for the requested resource path, and then return
> a
> > > JSON array with the possible mirror the client can use. Then, the
> client
> > > gets this list and does the processing required to select a mirror.
> > >
> > > Now let’s say that a human access the link using a browser. Then, the
> > > server should receive something like (text/plain) as the content-type;
> I
> > > think depending on the browser this field may differ a little (it would
> > > require some checking). Anyways, the server detects that it is a
> “human”
> > > requesting the resource, then we could serve an HTML page with a
> > Javascript
> > > that uses the mirror list. This Javascript could do some checking and
> > > choose the best mirror for that specific.
> > >
> > > Why do I think this approach is interesting?
> > > The application you developed would be used only to retrieve valid and
> > > trusted sources of system VM images (a trusted repository for mirrors
> > that
> > > we as a community vouch for).  Also, we shift the decision process
> > > regarding mirrors from the server to the client. Then, it is up to
> > clients
> > > to select mirrors, and not up to us (or some of our implementations).
> > >
> > > Did this help?
> > > BTW: I am not saying I am against the way you proposed, which would
> work
> > > fine. It is merely a suggestion using a different perspective.
> > >
> > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
> williamstevens@gmail.com>
> > > wrote:
> > >
> > > > @rafael: in general, I don't think the client should ever be making a
> > > > choice about a mirror. We have to assume we are working with a
> scripted
> > > > application and anything hard coded in that implementation on the
> > client
> > > > side is a risk.
> > > >
> > > > I may not be understanding the use of the 300 approach, so I will
> > > research
> > > > it to see if I can make it fit.
> > > >
> > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
> > > rafaelweingartner@gmail.com>
> > > > wrote:
> > > >
> > > > Will great job.
> > > >
> > > > I had the same doubt as Daan.
> > > >
> > > > Have you considered using HTTP 300 working mode? Then, we could let
> the
> > > > client decide which mirror is the best(closest?). Section “10.3.1” of
> > [1]
> > > > talks about it; this would put the pressure on deciding upon a mirror
> > on
> > > > the client side, but I think it would be fair.
> > > >
> > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
> > > >
> > > > PS: You really like very short variables names!
> > > >
> > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
> > > daan.hoogland@shapeblue.com
> > > > >
> > > > wrote:
> > > >
> > > > > Nice little thing Will,
> > > > >
> > > > > One question: if I read the code correctly it ‘go’es and tries all
> > > > mirrors
> > > > > at once and whichever responses first is redirected to the client.
> > This
> > > > > might well be the same every time. This might be the one closest to
> > the
> > > > > mirror302 and not to the client. Is that correct or did I miss a
> > > weighing
> > > > > algorithm hidden in there?
> > > > >
> > > > > Good coding,
> > > > >
> > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
> > > > >
> > > > >     Hey All,
> > > > >     Please review this repo and tell me what you think:
> > > > >     https://github.com/swill/mirror302
> > > > >
> > > > >     Let me know if you have questions or you would like me to make
> > > > changes
> > > > >     before I bring this topic up with ASF Infra and open a ticket
> to
> > > get
> > > > > this
> > > > >     implemented.
> > > > >
> > > > >     Cheers,
> > > > >
> > > > >     *Will Stevens*
> > > > >
> > > > >
> > > > >
> > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> > > wstevens@cloudops.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >     > I am building a short term solution right now.  I hopefully
> > will
> > > > > have the
> > > > >     > building blocks in place by the weekend so we can start
> working
> > > > with
> > > > > Infra
> > > > >     > to get it in place.
> > > > >     >
> > > > >     > We will have a web server which we can point something like
> > > > >     > downloads.cloudstack.org (or whatever url) at.  Then we will
> > > have
> > > > > an ASF
> > > > >     > repo which tracks a mirror list and exposes a static site to
> be
> > > > able
> > > > > to
> > > > >     > query the mirror list.
> > > > >     >
> > > > >     > The web server will receive a request and will do a 302
> > redirect
> > > to
> > > > > the
> > > > >     > appropriate resource on one of the mirrors.
> > > > >     >
> > > > >     > This gets us started.  Legacy environments will have to do a
> DB
> > > > > change to
> > > > >     > change from 'download.cloud.com' to '
> downloads.cloudstack.org'
> > > (or
> > > > >     > whatever), but otherwise it should be pretty simple.
> > > > >     >
> > > > >     > We can improve how we deliver templates going forward, but
> this
> > > > > obviously
> > > > >     > requires some discussion still.  I will try to buy us some
> time
> > > > with
> > > > > an
> > > > >     > implementation which solves our problems today...
> > > > >     >
> > > > >     > *Will STEVENS*
> > > > >     > Lead Developer
> > > > >     >
> > > > >     > <https://goo.gl/NYZ8KK>
> > > > >     >
> > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > > > >     > abhinandan.prateek@shapeblue.com> wrote:
> > > > >     >
> > > > >     >>
> > > > >     >> Initial seeding is a manual step and that is not going to
> > change
> > > > in
> > > > > near
> > > > >     >> future. A handy list of official places from where these
> > > templates
> > > > > can be
> > > > >     >> downloaded won’t harm this part of the installation. Just
> > allow
> > > > > anyone
> > > > >     >> credible who follows democratic process to publish their
> > > templates
> > > > > with
> > > > >     >> apache’s blessings.
> > > > >     >>
> > > > >     >>
> > > > >     >> Coming to urls hard coded in db, the popular suggestion is
> to
> > > > point
> > > > > them
> > > > >     >> to mirrors. This looks good but this is going to take some
> > > effort
> > > > > that has
> > > > >     >> not materialised yet. Once this change is made it will be
> > going
> > > in
> > > > > some
> > > > >     >> future release like 4.11; the problem still remains for the
> > > > > releases that
> > > > >     >> are already out there. In case access to download.cloud.com
> > is
> > > > > dropped
> > > > >     >> at some point we will have to document the procedure such
> that
> > > > > people know
> > > > >     >> how to make these stable releases work. That is where the
> > > > procedure
> > > > >     >> documented here https://shankerbalan.net/blog/
> > > > >     >> seed-cloudstack-templates-offline/ should be officially
> > adapted
> > > > > with a
> > > > >     >> marketplace to pick up templates of choosing. Maybe we can
> > have
> > > a
> > > > > pre-setup
> > > > >     >> script to make choices instead of written wiki. Note that
> this
> > > > will
> > > > > be
> > > > >     >> anyway needed by the older releases.
> > > > >     >>
> > > > >     >>
> > > > >     >>
> > > > >     >>
> > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
> paul.angus@shapeblue.com>
> > > > > wrote:
> > > > >     >>
> > > > >     >> >-1
> > > > >     >> >
> > > > >     >> >We are trying to get MORE people to use CloudStack.  And
> > > raising
> > > > > the
> > > > >     >> barrier to entry does nothing to help that.
> > > > >     >> >
> > > > >     >> >
> > > > >     >> >Kind regards,
> > > > >     >> >
> > > > >     >> >Paul Angus
> > > > >     >> >
> > > > >     >> >paul.angus@shapeblue.com
> > > > >     >> >www.shapeblue.com
> > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > >     >> >@shapeblue
> > > > >     >> >
> > > > >     >> >
> > > > >     >> >
> > > > >     >> >
> > > > >     >> >-----Original Message-----
> > > > >     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@
> > > > shapeblue.com]
> > > > >     >> >Sent: 01 March 2017 09:39
> > > > >     >> >To: dev@cloudstack.apache.org
> > > > >     >> >Subject: Re: Modern template hosting
> > > > >     >> >
> > > > >     >> >+1
> > > > >     >> >
> > > > >     >> >After seeing this discussion running in circles several
> > times,
> > > I
> > > > > think
> > > > >     >> we should at least get started with the simplest option.
> > > > >     >> >
> > > > >     >> >
> > > > >     >> >
> > > > >     >> >
> > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
> > > chiradeepv@gmail.com>
> > > > > wrote:
> > > > >     >> >
> > > > >     >> >>The alternative foolproof way is to simply not provide the
> > > > > automatic
> > > > >     >> >>download.
> > > > >     >> >>
> > > > >     >> >>Just document the mirror list URL where the templates can
> be
> > > > > found. I
> > > > >     >> >>don't know why people reject this option.
> > > > >     >> >>
> > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
> > > > templates-offline/
> > > > >     >> >>
> > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> > > > >     >> >><wi...@gmail.com>
> > > > >     >> >>wrote:
> > > > >     >> >>
> > > > >     >> >>> OK. Thanks for the heads up.
> > > > >     >> >>>
> > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > > chiradeepv@gmail.com>
> > > > >     >> wrote:
> > > > >     >> >>>
> > > > >     >> >>> > Sounds workable. The downloader code in the SSVM won't
> > > > follow
> > > > >     >> >>> > redirects I think.
> > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
> > > > 5511065fc20787619d
> > > > > 9cd0444
> > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
> > > > >     >> >>> > HttpTemplateDownloader.java#L93
> > > > >     >> >>> > https://goo.gl/dSi0r5
> > > > >     >> >>> >  Might need to add
> > > > >     >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > > >     >> >>> >
> > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> > > > >     >> >>> > <ws...@cloudops.com>
> > > > >     >> >>> > wrote:
> > > > >     >> >>> >
> > > > >     >> >>> > > We haven't opened a ticket yet because we don't
> have a
> > > > > strategy
> > > > >     >> yet.
> > > > >     >> >>> > >
> > > > >     >> >>> > > What do you guys think of this:
> > > > >     >> >>> > > - We setup a new github repo in the 'apache' org
> which
> > > > > consists
> > > > >     >> >>> > > of a
> > > > >     >> >>> > single
> > > > >     >> >>> > > file with a list of active/supported mirrors.
> > > > >     >> >>> > > - I write a small web server, distributed as a
> binary,
> > > > > which can
> > > > >     >> >>> > > be
> > > > >     >> >>> > hosted
> > > > >     >> >>> > > by ASF Infra.  This web server will query the
> current
> > > list
> > > > > of
> > > > >     >> >>> > > mirrors
> > > > >     >> >>> and
> > > > >     >> >>> > > will select one and then do a 302 redirect to that
> > > mirror.
> > > > >     >> >>> > >
> > > > >     >> >>> > > The act of 'choosing' a mirror could be done in a
> > number
> > > > of
> > > > > ways.
> > > > >     >> >>> > > - If we want to define an order, then it could just
> > try
> > > > > from the
> > > > >     >> >>> > > top of
> > > > >     >> >>> > the
> > > > >     >> >>> > > list and work its way down.  It would curl the
> target
> > to
> > > > > make
> > > > >     >> >>> > > sure it
> > > > >     >> >>> > gets
> > > > >     >> >>> > > a 200 and if it does, it would do a 302 redirect.
> > > > >     >> >>> > > - Or, if we want to distribute the load across the
> > > > mirrors,
> > > > > we
> > > > >     >> >>> > > could
> > > > >     >> >>> pick
> > > > >     >> >>> > > from the list randomly.  Again, doing a curl to
> verify
> > > the
> > > > > mirror
> > > > >     >> >>> > > is up
> > > > >     >> >>> > and
> > > > >     >> >>> > > then doing a redirect.
> > > > >     >> >>> > > - If we want to get fancy, we could do a reverse IP
> > > lookup
> > > > > and
> > > > >     >> >>> > > try to
> > > > >     >> >>> > match
> > > > >     >> >>> > > the requester with their closest geographical
> mirror.
> > > > >     >> >>> > >
> > > > >     >> >>> > > Thoughts?
> > > > >     >> >>> > >
> > > > >     >> >>> > > *Will STEVENS*
> > > > >     >> >>> > > Lead Developer
> > > > >     >> >>> > >
> > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
> > > > >     >> >>> > >
> > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > >     >> >>> chiradeepv@gmail.com
> > > > >     >> >>> > >
> > > > >     >> >>> > > wrote:
> > > > >     >> >>> > >
> > > > >     >> >>> > > > What steps are needed to set up a mirror? What
> does
> > > > Infra
> > > > > need
> > > > >     >> to do?
> > > > >     >> >>> > Has
> > > > >     >> >>> > > > anybody filed a ticket with Infra?
> > > > >     >> >>> > > >
> > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > >     >> >>> > > > raja.pullela@accelerite.com>
> > > > >     >> >>> > > > wrote:
> > > > >     >> >>> > > >
> > > > >     >> >>> > > > > Hi will,
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > I believe, we didn’t get to close ‘getting a
> > mirror
> > > on
> > > > > Apache’
> > > > >     >> >>> > because
> > > > >     >> >>> > > we
> > > > >     >> >>> > > > > needed someone on the Apache Infra side to close
> > > this.
> > > > > BTW,
> > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages
> this?)
> > > has
> > > > >     >> >>> > > > > all/most of
> > > > >     >> >>> > the
> > > > >     >> >>> > > > > content.  Once we can close on the Apache mirror
> > for
> > > > > hosting
> > > > >     >> >>> > > > > the
> > > > >     >> >>> > > > content, I
> > > > >     >> >>> > > > > can help assist getting the content there.
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > For now, we have replicated the
> > download.cloud.com
> > > > > content to
> > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
> > > > >     >> >>> > > > > Also, we are working on a set of steps/procedure
> > to
> > > > > help with
> > > > >     >> >>> > > > > this change.  I will update everyone in about a
> > > week’s
> > > > > time
> > > > >     >> >>> > > > > on the
> > > > >     >> >>> > details.
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > Best,
> > > > >     >> >>> > > > > Raja Pullela
> > > > >     >> >>> > > > > Engineering Team,
> > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara,
> CA,
> > > > 95054
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com
> > on
> > > > > behalf of
> > > > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com on
> behalf
> > > of
> > > > >     >> >>> > wstevens@cloudops.com>
> > > > >     >> >>> > > > > wrote:
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > unfortunately the template mirror conversation
> got
> > > > > caught up
> > > > >     >> >>> > > > > in
> > > > >     >> >>> > details
> > > > >     >> >>> > > > and
> > > > >     >> >>> > > > > nobody took the lead on implementing a solution.
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > citrix has been pinging me every couple months
> to
> > > say
> > > > > 'dude,
> > > > >     >> >>> > > > > we
> > > > >     >> >>> need
> > > > >     >> >>> > to
> > > > >     >> >>> > > > > remove the dependency on download.citrix.com',
> > but
> > > i
> > > > > have not
> > > > >     >> >>> > > > > had
> > > > >     >> >>> > the
> > > > >     >> >>> > > > > cycles to get in and solve the problem.  the
> > > shutdown
> > > > > of that
> > > > >     >> >>> > > > > is
> > > > >     >> >>> > > imminent
> > > > >     >> >>> > > > > right now, so we need to solve it asap.
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > *Will STEVENS*
> > > > >     >> >>> > > > > Lead Developer
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > >     >> >>> > paul.angus@shapeblue.com
> > > > >     >> >>> > > >
> > > > >     >> >>> > > > > wrote:
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > > Hi Nathan,
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > Ideally, if you put the template location in
> (or
> > > use
> > > > a
> > > > >     >> >>> > > > > > template
> > > > >     >> >>> > > defined
> > > > >     >> >>> > > > > > in)  test_data.py then the actual location can
> > be
> > > > >     >> >>> > > > > > overridden by
> > > > >     >> >>> > > anyone
> > > > >     >> >>> > > > > > testing.
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > For Trillian, we've copied all of the
> templates
> > > that
> > > > > people
> > > > >     >> >>> > > > > > have
> > > > >     >> >>> > > define
> > > > >     >> >>> > > > > to
> > > > >     >> >>> > > > > > a local repo and then replace the URLs in
> > > > > test_data.py to
> > > > >     >> >>> > > > > > reduce
> > > > >     >> >>> > > > > bandwidth
> > > > >     >> >>> > > > > > use and download times.
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > Ie:
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >             "bootableIso":
> > > > >     >> >>> > > > > >                 {
> > > > >     >> >>> > > > > >                     "displaytext": "Test
> > Bootable
> > > > > ISO",
> > > > >     >> >>> > > > > >                     "name": "testISO",
> > > > >     >> >>> > > > > >                     "bootable": True,
> > > > >     >> >>> > > > > >                     "ispublic": False,
> > > > >     >> >>> > > > > >                     "url": "{{
> > > > marvin_images_location
> > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
> > > > >     >> >>> > > > > >                     "ostype": 'Other Linux
> > > > (64-bit)',
> > > > >     >> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > >     >> >>> > > > > >         },
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > I thought that we had come up with a solution
> > for
> > > > >     >> >>> > download.cloud.com
> > > > >     >> >>> > > ,
> > > > >     >> >>> > > > by
> > > > >     >> >>> > > > > > having a mirrorlist hosted in Community Apache
> > > > > 'space' with
> > > > >     >> >>> anyone
> > > > >     >> >>> > > able
> > > > >     >> >>> > > > > to
> > > > >     >> >>> > > > > > out themselves forward as a mirror.
> > > > >     >> >>> > > > > > But I must admit I lost track of whether
> anyone
> > > made
> > > > > the
> > > > >     >> >>> requisite
> > > > >     >> >>> > > > > changes
> > > > >     >> >>> > > > > > in code....
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > Kind regards,
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > Paul Angus
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > paul.angus@shapeblue.com
> > > > >     >> >>> > > > > > www.shapeblue.com
> > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N
> > > 4HSUK
> > > > >     >> >>> > > > > > @shapeblue
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > -----Original Message-----
> > > > >     >> >>> > > > > > From: williamstevens@gmail.com
> > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
> > > > >     >> >>> > On
> > > > >     >> >>> > > > > > Behalf Of Will Stevens
> > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
> > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
> > > > >     >> >>> > > > > > Subject: Re: Modern template hosting
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > this is a hard questions.  in general, we
> should
> > > be
> > > > > setting
> > > > >     >> >>> > > > > > up a
> > > > >     >> >>> > > mirror
> > > > >     >> >>> > > > > on
> > > > >     >> >>> > > > > > some cloudstack/apache domain and then mirror
> to
> > > > other
> > > > >     >> >>> > > > > > provided
> > > > >     >> >>> > > > > templates.
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > we MUST come up with a solution to deprecate
> > > > >     >> >>> > > > > > 'download.cloud.com
> > > > >     >> >>> ',
> > > > >     >> >>> > > > that
> > > > >     >> >>> > > > > > is going to be going away any day now.
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > i don't know the right way to solve this to be
> > > > > honest, but
> > > > >     >> >>> > > > > > if you
> > > > >     >> >>> > > have
> > > > >     >> >>> > > > > > ideas, i am willing to help.
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > *Will STEVENS*
> > > > >     >> >>> > > > > > Lead Developer
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan
> > Johnson <
> > > > >     >> >>> njohnson@ena.com
> > > > >     >> >>> > >
> > > > >     >> >>> > > > > wrote:
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > > So not to re-open a can of worms, but I’m
> in a
> > > > > situation
> > > > >     >> >>> > > > > > > where
> > > > >     >> >>> I
> > > > >     >> >>> > > need
> > > > >     >> >>> > > > > > > to come up with a Marvin component test that
> > > > > depends on a
> > > > >     >> >>> > template
> > > > >     >> >>> > > > > > > based on a kernel that’s relatively new,
> i.e.,
> > > > > newer than
> > > > >     >> >>> Centos
> > > > >     >> >>> > > 5.3
> > > > >     >> >>> > > > /
> > > > >     >> >>> > > > > > Ubuntu 10.04 .
> > > > >     >> >>> > > > > > > I see openvm.eu has a suitable template
> > (Ubuntu
> > > > > 16.0.4
> > > > >     >> >>> > > > > > > for
> > > > >     >> >>> KVM),
> > > > >     >> >>> > > but
> > > > >     >> >>> > > > > > > from looking at the thread "Migrating
> > CloudStack
> > > > > content
> > > > >     >> >>> > > > > > > from download.cloud.com” it looks like
> there
> > is
> > > > >     >> >>> > > > > > > resistance to using
> > > > >     >> >>> > > this
> > > > >     >> >>> > > > at
> > > > >     >> >>> > > > > > > least for hosting system vm templates over
> > > > concerns
> > > > > of
> > > > >     >> >>> > neutrality.
> > > > >     >> >>> > > > > > > Would this be suitable for a component test?
> > If
> > > > > not,
> > > > >     >> >>> > > > > > > what is a
> > > > >     >> >>> > > > > “blessed”
> > > > >     >> >>> > > > > > > template location?
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > > Thanks in advance!
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > > Nathan Johnson
> > > > >     >> >>> > > > > > > R&D Engineer
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12
> Nashville,
> > TN
> > > > > 37211
> > > > >     >> >>> > > > > > > General Office: 615-312-6000
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > > website | blog | support
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > DISCLAIMER
> > > > >     >> >>> > > > > ==========
> > > > >     >> >>> > > > > This e-mail may contain privileged and
> > confidential
> > > > >     >> >>> > > > > information
> > > > >     >> >>> which
> > > > >     >> >>> > > is
> > > > >     >> >>> > > > > the property of Accelerite, a Persistent Systems
> > > > > business. It
> > > > >     >> >>> > > > > is
> > > > >     >> >>> > > intended
> > > > >     >> >>> > > > > only for the use of the individual or entity to
> > > which
> > > > > it is
> > > > >     >> >>> > addressed.
> > > > >     >> >>> > > If
> > > > >     >> >>> > > > > you are not the intended recipient, you are not
> > > > > authorized to
> > > > >     >> >>> > > > > read,
> > > > >     >> >>> > > > retain,
> > > > >     >> >>> > > > > copy, print, distribute or use this message. If
> > you
> > > > have
> > > > >     >> >>> > > > > received
> > > > >     >> >>> > this
> > > > >     >> >>> > > > > communication in error, please notify the sender
> > and
> > > > > delete
> > > > >     >> >>> > > > > all
> > > > >     >> >>> > copies
> > > > >     >> >>> > > of
> > > > >     >> >>> > > > > this message. Accelerite, a Persistent Systems
> > > > business
> > > > > does
> > > > >     >> >>> > > > > not
> > > > >     >> >>> > accept
> > > > >     >> >>> > > > any
> > > > >     >> >>> > > > > liability for virus infected mails.
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > >
> > > > >     >> >>> > >
> > > > >     >> >>> >
> > > > >     >> >>>
> > > > >     >> >
> > > > >     >> >abhinandan.prateek@shapeblue.com
> > > > >     >> >www.shapeblue.com
> > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> > > > >     >> >
> > > > >     >> >
> > > > >     >> >
> > > > >     >>
> > > > >     >> abhinandan.prateek@shapeblue.com
> > > > >     >> www.shapeblue.com
> > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > >     >> @shapeblue
> > > > >     >>
> > > > >     >>
> > > > >     >>
> > > > >     >>
> > > > >     >
> > > > >
> > > > >
> > > > >
> > > > > daan.hoogland@shapeblue.com
> > > > > www.shapeblue.com
> > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > @shapeblue
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Rafael Weingärtner
> > > >
> > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> > >
> >
>

Re: Modern template hosting

Posted by Chiradeep Vittal <ch...@gmail.com>.

1. If you are targeting legacy installations, they are not able to follow
redirects. The line of code that added this capability was added on 11/16.
2. If you trust the users to edit the database to change the URL, you can
trust them to change it to anything. Just document a known list of good
template locations.

On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <ws...@cloudops.com> wrote:

> So the main issue I see with this is the following.
>
> This implementation is designed to target legacy installations which will
> be affected when download.cloud.com disappears.  These people will need to
> make a DB change to replace the 'download.cloud.com' with some other url
> (maybe 'download.cloudstack.org').  Once the DB has been updated, we can
> not expect anything else of the client, they need to be able to just
> continue operation as they were without any need to rebuild or upgrade
> their ACS.
>
> If we try to force the decision to the client, then we break the legacy
> implementations.  Since the SSVM is likely going to be the client in some
> cases, and since it does not already have the logic to handle the 300
> approach correctly, I feel like it is not a viable solution for the legacy
> deployments.
>
> In addition to that.  I don't think the client is in any better position to
> make the mirror decision than I am.  I am likely able to make a more
> 'educated' decision than the client would be able to because I can do
> different tests on the endpoint(s) before making a final decision.
>
> Does this make sense or am I missing something obvious?
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
> rafaelweingartner@gmail.com> wrote:
>
> > I think I understood you, but I did not understand you.
> >
> > Let me see if I can create an example to illustrate how we could do using
> > the HTTP 300 code.
> >
> > The HTTP 300 code indicates clients that he/she/it has multiple choices
> to
> > access the requested resource. If the request is a HEAD, this is the only
> > thing that the server returns. If the request if a GET, then the server
> > returns the choices for the client. The response type is defined on the
> > server based on the Content-Type, user agent and others.
> >
> > Let’s say the server receives a GET request and informs the content-type
> as
> > JSON (application/JSON). Your application could get the mirrors list, do
> > the appending process for the requested resource path, and then return a
> > JSON array with the possible mirror the client can use. Then, the client
> > gets this list and does the processing required to select a mirror.
> >
> > Now let’s say that a human access the link using a browser. Then, the
> > server should receive something like (text/plain) as the content-type; I
> > think depending on the browser this field may differ a little (it would
> > require some checking). Anyways, the server detects that it is a “human”
> > requesting the resource, then we could serve an HTML page with a
> Javascript
> > that uses the mirror list. This Javascript could do some checking and
> > choose the best mirror for that specific.
> >
> > Why do I think this approach is interesting?
> > The application you developed would be used only to retrieve valid and
> > trusted sources of system VM images (a trusted repository for mirrors
> that
> > we as a community vouch for).  Also, we shift the decision process
> > regarding mirrors from the server to the client. Then, it is up to
> clients
> > to select mirrors, and not up to us (or some of our implementations).
> >
> > Did this help?
> > BTW: I am not saying I am against the way you proposed, which would work
> > fine. It is merely a suggestion using a different perspective.
> >
> > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <wi...@gmail.com>
> > wrote:
> >
> > > @rafael: in general, I don't think the client should ever be making a
> > > choice about a mirror. We have to assume we are working with a scripted
> > > application and anything hard coded in that implementation on the
> client
> > > side is a risk.
> > >
> > > I may not be understanding the use of the 300 approach, so I will
> > research
> > > it to see if I can make it fit.
> > >
> > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
> > rafaelweingartner@gmail.com>
> > > wrote:
> > >
> > > Will great job.
> > >
> > > I had the same doubt as Daan.
> > >
> > > Have you considered using HTTP 300 working mode? Then, we could let the
> > > client decide which mirror is the best(closest?). Section “10.3.1” of
> [1]
> > > talks about it; this would put the pressure on deciding upon a mirror
> on
> > > the client side, but I think it would be fair.
> > >
> > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
> > >
> > > PS: You really like very short variables names!
> > >
> > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
> > daan.hoogland@shapeblue.com
> > > >
> > > wrote:
> > >
> > > > Nice little thing Will,
> > > >
> > > > One question: if I read the code correctly it ‘go’es and tries all
> > > mirrors
> > > > at once and whichever responses first is redirected to the client.
> This
> > > > might well be the same every time. This might be the one closest to
> the
> > > > mirror302 and not to the client. Is that correct or did I miss a
> > weighing
> > > > algorithm hidden in there?
> > > >
> > > > Good coding,
> > > >
> > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
> > > >
> > > >     Hey All,
> > > >     Please review this repo and tell me what you think:
> > > >     https://github.com/swill/mirror302
> > > >
> > > >     Let me know if you have questions or you would like me to make
> > > changes
> > > >     before I bring this topic up with ASF Infra and open a ticket to
> > get
> > > > this
> > > >     implemented.
> > > >
> > > >     Cheers,
> > > >
> > > >     *Will Stevens*
> > > >
> > > >
> > > >
> > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> > wstevens@cloudops.com
> > > >
> > > > wrote:
> > > >
> > > >     > I am building a short term solution right now.  I hopefully
> will
> > > > have the
> > > >     > building blocks in place by the weekend so we can start working
> > > with
> > > > Infra
> > > >     > to get it in place.
> > > >     >
> > > >     > We will have a web server which we can point something like
> > > >     > downloads.cloudstack.org (or whatever url) at.  Then we will
> > have
> > > > an ASF
> > > >     > repo which tracks a mirror list and exposes a static site to be
> > > able
> > > > to
> > > >     > query the mirror list.
> > > >     >
> > > >     > The web server will receive a request and will do a 302
> redirect
> > to
> > > > the
> > > >     > appropriate resource on one of the mirrors.
> > > >     >
> > > >     > This gets us started.  Legacy environments will have to do a DB
> > > > change to
> > > >     > change from 'download.cloud.com' to 'downloads.cloudstack.org'
> > (or
> > > >     > whatever), but otherwise it should be pretty simple.
> > > >     >
> > > >     > We can improve how we deliver templates going forward, but this
> > > > obviously
> > > >     > requires some discussion still.  I will try to buy us some time
> > > with
> > > > an
> > > >     > implementation which solves our problems today...
> > > >     >
> > > >     > *Will STEVENS*
> > > >     > Lead Developer
> > > >     >
> > > >     > <https://goo.gl/NYZ8KK>
> > > >     >
> > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > > >     > abhinandan.prateek@shapeblue.com> wrote:
> > > >     >
> > > >     >>
> > > >     >> Initial seeding is a manual step and that is not going to
> change
> > > in
> > > > near
> > > >     >> future. A handy list of official places from where these
> > templates
> > > > can be
> > > >     >> downloaded won’t harm this part of the installation. Just
> allow
> > > > anyone
> > > >     >> credible who follows democratic process to publish their
> > templates
> > > > with
> > > >     >> apache’s blessings.
> > > >     >>
> > > >     >>
> > > >     >> Coming to urls hard coded in db, the popular suggestion is to
> > > point
> > > > them
> > > >     >> to mirrors. This looks good but this is going to take some
> > effort
> > > > that has
> > > >     >> not materialised yet. Once this change is made it will be
> going
> > in
> > > > some
> > > >     >> future release like 4.11; the problem still remains for the
> > > > releases that
> > > >     >> are already out there. In case access to download.cloud.com
> is
> > > > dropped
> > > >     >> at some point we will have to document the procedure such that
> > > > people know
> > > >     >> how to make these stable releases work. That is where the
> > > procedure
> > > >     >> documented here https://shankerbalan.net/blog/
> > > >     >> seed-cloudstack-templates-offline/ should be officially
> adapted
> > > > with a
> > > >     >> marketplace to pick up templates of choosing. Maybe we can
> have
> > a
> > > > pre-setup
> > > >     >> script to make choices instead of written wiki. Note that this
> > > will
> > > > be
> > > >     >> anyway needed by the older releases.
> > > >     >>
> > > >     >>
> > > >     >>
> > > >     >>
> > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <pa...@shapeblue.com>
> > > > wrote:
> > > >     >>
> > > >     >> >-1
> > > >     >> >
> > > >     >> >We are trying to get MORE people to use CloudStack.  And
> > raising
> > > > the
> > > >     >> barrier to entry does nothing to help that.
> > > >     >> >
> > > >     >> >
> > > >     >> >Kind regards,
> > > >     >> >
> > > >     >> >Paul Angus
> > > >     >> >
> > > >     >> >paul.angus@shapeblue.com
> > > >     >> >www.shapeblue.com
> > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > >     >> >@shapeblue
> > > >     >> >
> > > >     >> >
> > > >     >> >
> > > >     >> >
> > > >     >> >-----Original Message-----
> > > >     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@
> > > shapeblue.com]
> > > >     >> >Sent: 01 March 2017 09:39
> > > >     >> >To: dev@cloudstack.apache.org
> > > >     >> >Subject: Re: Modern template hosting
> > > >     >> >
> > > >     >> >+1
> > > >     >> >
> > > >     >> >After seeing this discussion running in circles several
> times,
> > I
> > > > think
> > > >     >> we should at least get started with the simplest option.
> > > >     >> >
> > > >     >> >
> > > >     >> >
> > > >     >> >
> > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
> > chiradeepv@gmail.com>
> > > > wrote:
> > > >     >> >
> > > >     >> >>The alternative foolproof way is to simply not provide the
> > > > automatic
> > > >     >> >>download.
> > > >     >> >>
> > > >     >> >>Just document the mirror list URL where the templates can be
> > > > found. I
> > > >     >> >>don't know why people reject this option.
> > > >     >> >>
> > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
> > > templates-offline/
> > > >     >> >>
> > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> > > >     >> >><wi...@gmail.com>
> > > >     >> >>wrote:
> > > >     >> >>
> > > >     >> >>> OK. Thanks for the heads up.
> > > >     >> >>>
> > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > chiradeepv@gmail.com>
> > > >     >> wrote:
> > > >     >> >>>
> > > >     >> >>> > Sounds workable. The downloader code in the SSVM won't
> > > follow
> > > >     >> >>> > redirects I think.
> > > >     >> >>> > https://github.com/apache/cloudstack/blob/
> > > 5511065fc20787619d
> > > > 9cd0444
> > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
> > > >     >> >>> > HttpTemplateDownloader.java#L93
> > > >     >> >>> > https://goo.gl/dSi0r5
> > > >     >> >>> >  Might need to add
> > > >     >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > >     >> >>> >
> > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> > > >     >> >>> > <ws...@cloudops.com>
> > > >     >> >>> > wrote:
> > > >     >> >>> >
> > > >     >> >>> > > We haven't opened a ticket yet because we don't have a
> > > > strategy
> > > >     >> yet.
> > > >     >> >>> > >
> > > >     >> >>> > > What do you guys think of this:
> > > >     >> >>> > > - We setup a new github repo in the 'apache' org which
> > > > consists
> > > >     >> >>> > > of a
> > > >     >> >>> > single
> > > >     >> >>> > > file with a list of active/supported mirrors.
> > > >     >> >>> > > - I write a small web server, distributed as a binary,
> > > > which can
> > > >     >> >>> > > be
> > > >     >> >>> > hosted
> > > >     >> >>> > > by ASF Infra.  This web server will query the current
> > list
> > > > of
> > > >     >> >>> > > mirrors
> > > >     >> >>> and
> > > >     >> >>> > > will select one and then do a 302 redirect to that
> > mirror.
> > > >     >> >>> > >
> > > >     >> >>> > > The act of 'choosing' a mirror could be done in a
> number
> > > of
> > > > ways.
> > > >     >> >>> > > - If we want to define an order, then it could just
> try
> > > > from the
> > > >     >> >>> > > top of
> > > >     >> >>> > the
> > > >     >> >>> > > list and work its way down.  It would curl the target
> to
> > > > make
> > > >     >> >>> > > sure it
> > > >     >> >>> > gets
> > > >     >> >>> > > a 200 and if it does, it would do a 302 redirect.
> > > >     >> >>> > > - Or, if we want to distribute the load across the
> > > mirrors,
> > > > we
> > > >     >> >>> > > could
> > > >     >> >>> pick
> > > >     >> >>> > > from the list randomly.  Again, doing a curl to verify
> > the
> > > > mirror
> > > >     >> >>> > > is up
> > > >     >> >>> > and
> > > >     >> >>> > > then doing a redirect.
> > > >     >> >>> > > - If we want to get fancy, we could do a reverse IP
> > lookup
> > > > and
> > > >     >> >>> > > try to
> > > >     >> >>> > match
> > > >     >> >>> > > the requester with their closest geographical mirror.
> > > >     >> >>> > >
> > > >     >> >>> > > Thoughts?
> > > >     >> >>> > >
> > > >     >> >>> > > *Will STEVENS*
> > > >     >> >>> > > Lead Developer
> > > >     >> >>> > >
> > > >     >> >>> > > <https://goo.gl/NYZ8KK>
> > > >     >> >>> > >
> > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > >     >> >>> chiradeepv@gmail.com
> > > >     >> >>> > >
> > > >     >> >>> > > wrote:
> > > >     >> >>> > >
> > > >     >> >>> > > > What steps are needed to set up a mirror? What does
> > > Infra
> > > > need
> > > >     >> to do?
> > > >     >> >>> > Has
> > > >     >> >>> > > > anybody filed a ticket with Infra?
> > > >     >> >>> > > >
> > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > >     >> >>> > > > raja.pullela@accelerite.com>
> > > >     >> >>> > > > wrote:
> > > >     >> >>> > > >
> > > >     >> >>> > > > > Hi will,
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > I believe, we didn’t get to close ‘getting a
> mirror
> > on
> > > > Apache’
> > > >     >> >>> > because
> > > >     >> >>> > > we
> > > >     >> >>> > > > > needed someone on the Apache Infra side to close
> > this.
> > > > BTW,
> > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages this?)
> > has
> > > >     >> >>> > > > > all/most of
> > > >     >> >>> > the
> > > >     >> >>> > > > > content.  Once we can close on the Apache mirror
> for
> > > > hosting
> > > >     >> >>> > > > > the
> > > >     >> >>> > > > content, I
> > > >     >> >>> > > > > can help assist getting the content there.
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > For now, we have replicated the
> download.cloud.com
> > > > content to
> > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
> > > >     >> >>> > > > > Also, we are working on a set of steps/procedure
> to
> > > > help with
> > > >     >> >>> > > > > this change.  I will update everyone in about a
> > week’s
> > > > time
> > > >     >> >>> > > > > on the
> > > >     >> >>> > details.
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > Best,
> > > >     >> >>> > > > > Raja Pullela
> > > >     >> >>> > > > > Engineering Team,
> > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA,
> > > 95054
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com
> on
> > > > behalf of
> > > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com on behalf
> > of
> > > >     >> >>> > wstevens@cloudops.com>
> > > >     >> >>> > > > > wrote:
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > unfortunately the template mirror conversation got
> > > > caught up
> > > >     >> >>> > > > > in
> > > >     >> >>> > details
> > > >     >> >>> > > > and
> > > >     >> >>> > > > > nobody took the lead on implementing a solution.
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > citrix has been pinging me every couple months to
> > say
> > > > 'dude,
> > > >     >> >>> > > > > we
> > > >     >> >>> need
> > > >     >> >>> > to
> > > >     >> >>> > > > > remove the dependency on download.citrix.com',
> but
> > i
> > > > have not
> > > >     >> >>> > > > > had
> > > >     >> >>> > the
> > > >     >> >>> > > > > cycles to get in and solve the problem.  the
> > shutdown
> > > > of that
> > > >     >> >>> > > > > is
> > > >     >> >>> > > imminent
> > > >     >> >>> > > > > right now, so we need to solve it asap.
> > > >     >> >>> > > > >
> > > >     >> >>> > > > >
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > *Will STEVENS*
> > > >     >> >>> > > > > Lead Developer
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > >     >> >>> > paul.angus@shapeblue.com
> > > >     >> >>> > > >
> > > >     >> >>> > > > > wrote:
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > > Hi Nathan,
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > Ideally, if you put the template location in (or
> > use
> > > a
> > > >     >> >>> > > > > > template
> > > >     >> >>> > > defined
> > > >     >> >>> > > > > > in)  test_data.py then the actual location can
> be
> > > >     >> >>> > > > > > overridden by
> > > >     >> >>> > > anyone
> > > >     >> >>> > > > > > testing.
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > For Trillian, we've copied all of the templates
> > that
> > > > people
> > > >     >> >>> > > > > > have
> > > >     >> >>> > > define
> > > >     >> >>> > > > > to
> > > >     >> >>> > > > > > a local repo and then replace the URLs in
> > > > test_data.py to
> > > >     >> >>> > > > > > reduce
> > > >     >> >>> > > > > bandwidth
> > > >     >> >>> > > > > > use and download times.
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > Ie:
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >             "bootableIso":
> > > >     >> >>> > > > > >                 {
> > > >     >> >>> > > > > >                     "displaytext": "Test
> Bootable
> > > > ISO",
> > > >     >> >>> > > > > >                     "name": "testISO",
> > > >     >> >>> > > > > >                     "bootable": True,
> > > >     >> >>> > > > > >                     "ispublic": False,
> > > >     >> >>> > > > > >                     "url": "{{
> > > marvin_images_location
> > > >     >> >>> > > > > > }}/TinyCore-current.iso",
> > > >     >> >>> > > > > >                     "ostype": 'Other Linux
> > > (64-bit)',
> > > >     >> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > >     >> >>> > > > > >         },
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > I thought that we had come up with a solution
> for
> > > >     >> >>> > download.cloud.com
> > > >     >> >>> > > ,
> > > >     >> >>> > > > by
> > > >     >> >>> > > > > > having a mirrorlist hosted in Community Apache
> > > > 'space' with
> > > >     >> >>> anyone
> > > >     >> >>> > > able
> > > >     >> >>> > > > > to
> > > >     >> >>> > > > > > out themselves forward as a mirror.
> > > >     >> >>> > > > > > But I must admit I lost track of whether anyone
> > made
> > > > the
> > > >     >> >>> requisite
> > > >     >> >>> > > > > changes
> > > >     >> >>> > > > > > in code....
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > Kind regards,
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > Paul Angus
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > paul.angus@shapeblue.com
> > > >     >> >>> > > > > > www.shapeblue.com
> > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N
> > 4HSUK
> > > >     >> >>> > > > > > @shapeblue
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > -----Original Message-----
> > > >     >> >>> > > > > > From: williamstevens@gmail.com
> > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
> > > >     >> >>> > On
> > > >     >> >>> > > > > > Behalf Of Will Stevens
> > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
> > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
> > > >     >> >>> > > > > > Subject: Re: Modern template hosting
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > this is a hard questions.  in general, we should
> > be
> > > > setting
> > > >     >> >>> > > > > > up a
> > > >     >> >>> > > mirror
> > > >     >> >>> > > > > on
> > > >     >> >>> > > > > > some cloudstack/apache domain and then mirror to
> > > other
> > > >     >> >>> > > > > > provided
> > > >     >> >>> > > > > templates.
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > we MUST come up with a solution to deprecate
> > > >     >> >>> > > > > > 'download.cloud.com
> > > >     >> >>> ',
> > > >     >> >>> > > > that
> > > >     >> >>> > > > > > is going to be going away any day now.
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > i don't know the right way to solve this to be
> > > > honest, but
> > > >     >> >>> > > > > > if you
> > > >     >> >>> > > have
> > > >     >> >>> > > > > > ideas, i am willing to help.
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > *Will STEVENS*
> > > >     >> >>> > > > > > Lead Developer
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan
> Johnson <
> > > >     >> >>> njohnson@ena.com
> > > >     >> >>> > >
> > > >     >> >>> > > > > wrote:
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > > So not to re-open a can of worms, but I’m in a
> > > > situation
> > > >     >> >>> > > > > > > where
> > > >     >> >>> I
> > > >     >> >>> > > need
> > > >     >> >>> > > > > > > to come up with a Marvin component test that
> > > > depends on a
> > > >     >> >>> > template
> > > >     >> >>> > > > > > > based on a kernel that’s relatively new, i.e.,
> > > > newer than
> > > >     >> >>> Centos
> > > >     >> >>> > > 5.3
> > > >     >> >>> > > > /
> > > >     >> >>> > > > > > Ubuntu 10.04 .
> > > >     >> >>> > > > > > > I see openvm.eu has a suitable template
> (Ubuntu
> > > > 16.0.4
> > > >     >> >>> > > > > > > for
> > > >     >> >>> KVM),
> > > >     >> >>> > > but
> > > >     >> >>> > > > > > > from looking at the thread "Migrating
> CloudStack
> > > > content
> > > >     >> >>> > > > > > > from download.cloud.com” it looks like there
> is
> > > >     >> >>> > > > > > > resistance to using
> > > >     >> >>> > > this
> > > >     >> >>> > > > at
> > > >     >> >>> > > > > > > least for hosting system vm templates over
> > > concerns
> > > > of
> > > >     >> >>> > neutrality.
> > > >     >> >>> > > > > > > Would this be suitable for a component test?
> If
> > > > not,
> > > >     >> >>> > > > > > > what is a
> > > >     >> >>> > > > > “blessed”
> > > >     >> >>> > > > > > > template location?
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > > Thanks in advance!
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > > Nathan Johnson
> > > >     >> >>> > > > > > > R&D Engineer
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville,
> TN
> > > > 37211
> > > >     >> >>> > > > > > > General Office: 615-312-6000
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > > website | blog | support
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > >
> > > >     >> >>> > > > >
> > > >     >> >>> > > > >
> > > >     >> >>> > > > >
> > > >     >> >>> > > > >
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > DISCLAIMER
> > > >     >> >>> > > > > ==========
> > > >     >> >>> > > > > This e-mail may contain privileged and
> confidential
> > > >     >> >>> > > > > information
> > > >     >> >>> which
> > > >     >> >>> > > is
> > > >     >> >>> > > > > the property of Accelerite, a Persistent Systems
> > > > business. It
> > > >     >> >>> > > > > is
> > > >     >> >>> > > intended
> > > >     >> >>> > > > > only for the use of the individual or entity to
> > which
> > > > it is
> > > >     >> >>> > addressed.
> > > >     >> >>> > > If
> > > >     >> >>> > > > > you are not the intended recipient, you are not
> > > > authorized to
> > > >     >> >>> > > > > read,
> > > >     >> >>> > > > retain,
> > > >     >> >>> > > > > copy, print, distribute or use this message. If
> you
> > > have
> > > >     >> >>> > > > > received
> > > >     >> >>> > this
> > > >     >> >>> > > > > communication in error, please notify the sender
> and
> > > > delete
> > > >     >> >>> > > > > all
> > > >     >> >>> > copies
> > > >     >> >>> > > of
> > > >     >> >>> > > > > this message. Accelerite, a Persistent Systems
> > > business
> > > > does
> > > >     >> >>> > > > > not
> > > >     >> >>> > accept
> > > >     >> >>> > > > any
> > > >     >> >>> > > > > liability for virus infected mails.
> > > >     >> >>> > > > >
> > > >     >> >>> > > >
> > > >     >> >>> > >
> > > >     >> >>> >
> > > >     >> >>>
> > > >     >> >
> > > >     >> >abhinandan.prateek@shapeblue.com
> > > >     >> >www.shapeblue.com
> > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
> > > >     >> >
> > > >     >> >
> > > >     >> >
> > > >     >>
> > > >     >> abhinandan.prateek@shapeblue.com
> > > >     >> www.shapeblue.com
> > > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > >     >> @shapeblue
> > > >     >>
> > > >     >>
> > > >     >>
> > > >     >>
> > > >     >
> > > >
> > > >
> > > >
> > > > daan.hoogland@shapeblue.com
> > > > www.shapeblue.com
> > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > @shapeblue
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> > >
> >
> >
> >
> > --
> > Rafael Weingärtner
> >
>

Re: Modern template hosting

Posted by Will Stevens <ws...@cloudops.com>.

ws: Inline...

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Fri, Mar 3, 2017 at 12:01 PM, Rafael Weingärtner <
rafaelweingartner@gmail.com> wrote:

> Yes it makes sense. I agree with you regarding the legacy application. I
> forgot them, my bad. Considering legacy application, I do agree that using
> the HTTP 300 method will not work.
>

ws: Ya, I think we have to keep this in mind as it is the most pressing
issue that needs to be solved.

>
> Now regarding the second point in the discussion; I think clients have a
> better position to decide, which mirror is the best for them. The same
> process you do to check the mirror latency, or check if the mirror is
> online could be executed on the client side. As I said before, this shifts
> the responsibility to the client side (this may be a good or a bad thing
> depending on the perspective).
>

ws: ASSUMING the client scripts a solution similar to what I have done
with the HEAD request on the resource and they pick the one with the lowest
latency, then yes, I agree that is a better solution.  If they are just
randomly picking one of the mirrors returned, then no, I don't think it is
better than what I can provide from the server.

The interesting thing here is that we don't have to choose only one
approach if I am implementing this web server.  We COULD introduce the
concept of using Headers or Query Params to give the client a choice of
how they want the mirror to be chosen.  In that case, we would want a
simple fail safe method to be the default and then the more 'advanced'
options to be handled by padding additional information.  This way the
client can choose the 300 approach if the are setup to consume and make an
educated decision about the results.

Something like this:
a) Default: use the current implementation since it handles loaded mirrors
pretty well.
b) Passing a flag would make the server respond with the 300 implementation.
c) Passing a flag would make the server chose the closest geographically
located mirror which successfully responded to the HEAD request.

Does this make sense?

>
> Anyways, I do not think that it would be feasible to use the approach I
> proposed given legacy systems. When I proposed, I did not think of this
> problem.
>
> Besides that comment, I think the system is ok ;)
>
> Before, you also asked opinions about the use of “cloudstack-www” as a repo
> for this mirror list file. I am in favor of that. Then, it would not
> require setting up other repositories and routines to get data from Github
> to a web server at Apache.
>

ws: Ya, I thought of this while walking home from the train and I think it
makes the most sense.  We already have everything in place to make it work.

>
> On Fri, Mar 3, 2017 at 11:13 AM, Will Stevens <ws...@cloudops.com>
> wrote:
>
> > So the main issue I see with this is the following.
> >
> > This implementation is designed to target legacy installations which will
> > be affected when download.cloud.com disappears.  These people will need
> to
> > make a DB change to replace the 'download.cloud.com' with some other url
> > (maybe 'download.cloudstack.org').  Once the DB has been updated, we can
> > not expect anything else of the client, they need to be able to just
> > continue operation as they were without any need to rebuild or upgrade
> > their ACS.
> >
> > If we try to force the decision to the client, then we break the legacy
> > implementations.  Since the SSVM is likely going to be the client in some
> > cases, and since it does not already have the logic to handle the 300
> > approach correctly, I feel like it is not a viable solution for the
> legacy
> > deployments.
> >
> > In addition to that.  I don't think the client is in any better position
> to
> > make the mirror decision than I am.  I am likely able to make a more
> > 'educated' decision than the client would be able to because I can do
> > different tests on the endpoint(s) before making a final decision.
> >
> > Does this make sense or am I missing something obvious?
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
> > rafaelweingartner@gmail.com> wrote:
> >
> > > I think I understood you, but I did not understand you.
> > >
> > > Let me see if I can create an example to illustrate how we could do
> using
> > > the HTTP 300 code.
> > >
> > > The HTTP 300 code indicates clients that he/she/it has multiple choices
> > to
> > > access the requested resource. If the request is a HEAD, this is the
> only
> > > thing that the server returns. If the request if a GET, then the server
> > > returns the choices for the client. The response type is defined on the
> > > server based on the Content-Type, user agent and others.
> > >
> > > Let’s say the server receives a GET request and informs the
> content-type
> > as
> > > JSON (application/JSON). Your application could get the mirrors list,
> do
> > > the appending process for the requested resource path, and then return
> a
> > > JSON array with the possible mirror the client can use. Then, the
> client
> > > gets this list and does the processing required to select a mirror.
> > >
> > > Now let’s say that a human access the link using a browser. Then, the
> > > server should receive something like (text/plain) as the content-type;
> I
> > > think depending on the browser this field may differ a little (it would
> > > require some checking). Anyways, the server detects that it is a
> “human”
> > > requesting the resource, then we could serve an HTML page with a
> > Javascript
> > > that uses the mirror list. This Javascript could do some checking and
> > > choose the best mirror for that specific.
> > >
> > > Why do I think this approach is interesting?
> > > The application you developed would be used only to retrieve valid and
> > > trusted sources of system VM images (a trusted repository for mirrors
> > that
> > > we as a community vouch for).  Also, we shift the decision process
> > > regarding mirrors from the server to the client. Then, it is up to
> > clients
> > > to select mirrors, and not up to us (or some of our implementations).
> > >
> > > Did this help?
> > > BTW: I am not saying I am against the way you proposed, which would
> work
> > > fine. It is merely a suggestion using a different perspective.
> > >
> > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <
> williamstevens@gmail.com>
> > > wrote:
> > >
> > > > @rafael: in general, I don't think the client should ever be making a
> > > > choice about a mirror. We have to assume we are working with a
> scripted
> > > > application and anything hard coded in that implementation on the
> > client
> > > > side is a risk.
> > > >
> > > > I may not be understanding the use of the 300 approach, so I will
> > > research
> > > > it to see if I can make it fit.
> > > >
> > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
> > > rafaelweingartner@gmail.com>
> > > > wrote:
> > > >
> > > > Will great job.
> > > >
> > > > I had the same doubt as Daan.
> > > >
> > > > Have you considered using HTTP 300 working mode? Then, we could let
> the
> > > > client decide which mirror is the best(closest?). Section “10.3.1” of
> > [1]
> > > > talks about it; this would put the pressure on deciding upon a mirror
> > on
> > > > the client side, but I think it would be fair.
> > > >
> > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
> > > >
> > > > PS: You really like very short variables names!
> > > >
> > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
> > > daan.hoogland@shapeblue.com
> > > > >
> > > > wrote:
> > > >
> > > > > Nice little thing Will,
> > > > >
> > > > > One question: if I read the code correctly it ‘go’es and tries all
> > > > mirrors
> > > > > at once and whichever responses first is redirected to the client.
> > This
> > > > > might well be the same every time. This might be the one closest to
> > the
> > > > > mirror302 and not to the client. Is that correct or did I miss a
> > > weighing
> > > > > algorithm hidden in there?
> > > > >
> > > > > Good coding,
> > > > >
> > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
> > > > >
> > > > >     Hey All,
> > > > >     Please review this repo and tell me what you think:
> > > > >     https://github.com/swill/mirror302
> > > > >
> > > > >     Let me know if you have questions or you would like me to make
> > > > changes
> > > > >     before I bring this topic up with ASF Infra and open a ticket
> to
> > > get
> > > > > this
> > > > >     implemented.
> > > > >
> > > > >     Cheers,
> > > > >
> > > > >     *Will Stevens*
> > > > >
> > > > >
> > > > >
> > > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> > > wstevens@cloudops.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >     > I am building a short term solution right now.  I hopefully
> > will
> > > > > have the
> > > > >     > building blocks in place by the weekend so we can start
> working
> > > > with
> > > > > Infra
> > > > >     > to get it in place.
> > > > >     >
> > > > >     > We will have a web server which we can point something like
> > > > >     > downloads.cloudstack.org (or whatever url) at.  Then we will
> > > have
> > > > > an ASF
> > > > >     > repo which tracks a mirror list and exposes a static site to
> be
> > > > able
> > > > > to
> > > > >     > query the mirror list.
> > > > >     >
> > > > >     > The web server will receive a request and will do a 302
> > redirect
> > > to
> > > > > the
> > > > >     > appropriate resource on one of the mirrors.
> > > > >     >
> > > > >     > This gets us started.  Legacy environments will have to do a
> DB
> > > > > change to
> > > > >     > change from 'download.cloud.com' to '
> downloads.cloudstack.org'
> > > (or
> > > > >     > whatever), but otherwise it should be pretty simple.
> > > > >     >
> > > > >     > We can improve how we deliver templates going forward, but
> this
> > > > > obviously
> > > > >     > requires some discussion still.  I will try to buy us some
> time
> > > > with
> > > > > an
> > > > >     > implementation which solves our problems today...
> > > > >     >
> > > > >     > *Will STEVENS*
> > > > >     > Lead Developer
> > > > >     >
> > > > >     > <https://goo.gl/NYZ8KK>
> > > > >     >
> > > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > > > >     > abhinandan.prateek@shapeblue.com> wrote:
> > > > >     >
> > > > >     >>
> > > > >     >> Initial seeding is a manual step and that is not going to
> > change
> > > > in
> > > > > near
> > > > >     >> future. A handy list of official places from where these
> > > templates
> > > > > can be
> > > > >     >> downloaded won’t harm this part of the installation. Just
> > allow
> > > > > anyone
> > > > >     >> credible who follows democratic process to publish their
> > > templates
> > > > > with
> > > > >     >> apache’s blessings.
> > > > >     >>
> > > > >     >>
> > > > >     >> Coming to urls hard coded in db, the popular suggestion is
> to
> > > > point
> > > > > them
> > > > >     >> to mirrors. This looks good but this is going to take some
> > > effort
> > > > > that has
> > > > >     >> not materialised yet. Once this change is made it will be
> > going
> > > in
> > > > > some
> > > > >     >> future release like 4.11; the problem still remains for the
> > > > > releases that
> > > > >     >> are already out there. In case access to download.cloud.com
> > is
> > > > > dropped
> > > > >     >> at some point we will have to document the procedure such
> that
> > > > > people know
> > > > >     >> how to make these stable releases work. That is where the
> > > > procedure
> > > > >     >> documented here https://shankerbalan.net/blog/
> > > > >     >> seed-cloudstack-templates-offline/ should be officially
> > adapted
> > > > > with a
> > > > >     >> marketplace to pick up templates of choosing. Maybe we can
> > have
> > > a
> > > > > pre-setup
> > > > >     >> script to make choices instead of written wiki. Note that
> this
> > > > will
> > > > > be
> > > > >     >> anyway needed by the older releases.
> > > > >     >>
> > > > >     >>
> > > > >     >>
> > > > >     >>
> > > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <
> paul.angus@shapeblue.com>
> > > > > wrote:
> > > > >     >>
> > > > >     >> >-1
> > > > >     >> >
> > > > >     >> >We are trying to get MORE people to use CloudStack.  And
> > > raising
> > > > > the
> > > > >     >> barrier to entry does nothing to help that.
> > > > >     >> >
> > > > >     >> >
> > > > >     >> >Kind regards,
> > > > >     >> >
> > > > >     >> >Paul Angus
> > > > >     >> >
> > > > >     >> >paul.angus@shapeblue.com
> > > > >     >> >www.shapeblue.com
> > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > >     >> >@shapeblue
> > > > >     >> >
> > > > >     >> >
> > > > >     >> >
> > > > >     >> >
> > > > >     >> >-----Original Message-----
> > > > >     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@
> > > > shapeblue.com]
> > > > >     >> >Sent: 01 March 2017 09:39
> > > > >     >> >To: dev@cloudstack.apache.org
> > > > >     >> >Subject: Re: Modern template hosting
> > > > >     >> >
> > > > >     >> >+1
> > > > >     >> >
> > > > >     >> >After seeing this discussion running in circles several
> > times,
> > > I
> > > > > think
> > > > >     >> we should at least get started with the simplest option.
> > > > >     >> >
> > > > >     >> >
> > > > >     >> >
> > > > >     >> >
> > > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
> > > chiradeepv@gmail.com>
> > > > > wrote:
> > > > >     >> >
> > > > >     >> >>The alternative foolproof way is to simply not provide the
> > > > > automatic
> > > > >     >> >>download.
> > > > >     >> >>
> > > > >     >> >>Just document the mirror list URL where the templates can
> be
> > > > > found. I
> > > > >     >> >>don't know why people reject this option.
> > > > >     >> >>
> > > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
> > > > templates-offline/
> > > > >     >> >>
> > > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> > > > >     >> >><wi...@gmail.com>
> > > > >     >> >>wrote:
> > > > >     >> >>
> > > > >     >> >>> OK. Thanks for the heads up.
> > > > >     >> >>>
> > > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > > chiradeepv@gmail.com>
> > > > >     >> wrote:
> > > > >     >> >>>
> > > > >     >> >>> > Sounds workable. The downloader code in the SSVM won't
> > > > follow
> > > > >     >> >>> > redirects I think.
> > > > >     >> >>> > https://github.com/apache/cloudstack/blob/
> > > > 5511065fc20787619d
> > > > > 9cd0444
> > > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
> > > > >     >> >>> > HttpTemplateDownloader.java#L93
> > > > >     >> >>> > https://goo.gl/dSi0r5
> > > > >     >> >>> >  Might need to add
> > > > >     >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > > >     >> >>> >
> > > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> > > > >     >> >>> > <ws...@cloudops.com>
> > > > >     >> >>> > wrote:
> > > > >     >> >>> >
> > > > >     >> >>> > > We haven't opened a ticket yet because we don't
> have a
> > > > > strategy
> > > > >     >> yet.
> > > > >     >> >>> > >
> > > > >     >> >>> > > What do you guys think of this:
> > > > >     >> >>> > > - We setup a new github repo in the 'apache' org
> which
> > > > > consists
> > > > >     >> >>> > > of a
> > > > >     >> >>> > single
> > > > >     >> >>> > > file with a list of active/supported mirrors.
> > > > >     >> >>> > > - I write a small web server, distributed as a
> binary,
> > > > > which can
> > > > >     >> >>> > > be
> > > > >     >> >>> > hosted
> > > > >     >> >>> > > by ASF Infra.  This web server will query the
> current
> > > list
> > > > > of
> > > > >     >> >>> > > mirrors
> > > > >     >> >>> and
> > > > >     >> >>> > > will select one and then do a 302 redirect to that
> > > mirror.
> > > > >     >> >>> > >
> > > > >     >> >>> > > The act of 'choosing' a mirror could be done in a
> > number
> > > > of
> > > > > ways.
> > > > >     >> >>> > > - If we want to define an order, then it could just
> > try
> > > > > from the
> > > > >     >> >>> > > top of
> > > > >     >> >>> > the
> > > > >     >> >>> > > list and work its way down.  It would curl the
> target
> > to
> > > > > make
> > > > >     >> >>> > > sure it
> > > > >     >> >>> > gets
> > > > >     >> >>> > > a 200 and if it does, it would do a 302 redirect.
> > > > >     >> >>> > > - Or, if we want to distribute the load across the
> > > > mirrors,
> > > > > we
> > > > >     >> >>> > > could
> > > > >     >> >>> pick
> > > > >     >> >>> > > from the list randomly.  Again, doing a curl to
> verify
> > > the
> > > > > mirror
> > > > >     >> >>> > > is up
> > > > >     >> >>> > and
> > > > >     >> >>> > > then doing a redirect.
> > > > >     >> >>> > > - If we want to get fancy, we could do a reverse IP
> > > lookup
> > > > > and
> > > > >     >> >>> > > try to
> > > > >     >> >>> > match
> > > > >     >> >>> > > the requester with their closest geographical
> mirror.
> > > > >     >> >>> > >
> > > > >     >> >>> > > Thoughts?
> > > > >     >> >>> > >
> > > > >     >> >>> > > *Will STEVENS*
> > > > >     >> >>> > > Lead Developer
> > > > >     >> >>> > >
> > > > >     >> >>> > > <https://goo.gl/NYZ8KK>
> > > > >     >> >>> > >
> > > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > >     >> >>> chiradeepv@gmail.com
> > > > >     >> >>> > >
> > > > >     >> >>> > > wrote:
> > > > >     >> >>> > >
> > > > >     >> >>> > > > What steps are needed to set up a mirror? What
> does
> > > > Infra
> > > > > need
> > > > >     >> to do?
> > > > >     >> >>> > Has
> > > > >     >> >>> > > > anybody filed a ticket with Infra?
> > > > >     >> >>> > > >
> > > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > >     >> >>> > > > raja.pullela@accelerite.com>
> > > > >     >> >>> > > > wrote:
> > > > >     >> >>> > > >
> > > > >     >> >>> > > > > Hi will,
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > I believe, we didn’t get to close ‘getting a
> > mirror
> > > on
> > > > > Apache’
> > > > >     >> >>> > because
> > > > >     >> >>> > > we
> > > > >     >> >>> > > > > needed someone on the Apache Infra side to close
> > > this.
> > > > > BTW,
> > > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages
> this?)
> > > has
> > > > >     >> >>> > > > > all/most of
> > > > >     >> >>> > the
> > > > >     >> >>> > > > > content.  Once we can close on the Apache mirror
> > for
> > > > > hosting
> > > > >     >> >>> > > > > the
> > > > >     >> >>> > > > content, I
> > > > >     >> >>> > > > > can help assist getting the content there.
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > For now, we have replicated the
> > download.cloud.com
> > > > > content to
> > > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
> > > > >     >> >>> > > > > Also, we are working on a set of steps/procedure
> > to
> > > > > help with
> > > > >     >> >>> > > > > this change.  I will update everyone in about a
> > > week’s
> > > > > time
> > > > >     >> >>> > > > > on the
> > > > >     >> >>> > details.
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > Best,
> > > > >     >> >>> > > > > Raja Pullela
> > > > >     >> >>> > > > > Engineering Team,
> > > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara,
> CA,
> > > > 95054
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com
> > on
> > > > > behalf of
> > > > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com on
> behalf
> > > of
> > > > >     >> >>> > wstevens@cloudops.com>
> > > > >     >> >>> > > > > wrote:
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > unfortunately the template mirror conversation
> got
> > > > > caught up
> > > > >     >> >>> > > > > in
> > > > >     >> >>> > details
> > > > >     >> >>> > > > and
> > > > >     >> >>> > > > > nobody took the lead on implementing a solution.
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > citrix has been pinging me every couple months
> to
> > > say
> > > > > 'dude,
> > > > >     >> >>> > > > > we
> > > > >     >> >>> need
> > > > >     >> >>> > to
> > > > >     >> >>> > > > > remove the dependency on download.citrix.com',
> > but
> > > i
> > > > > have not
> > > > >     >> >>> > > > > had
> > > > >     >> >>> > the
> > > > >     >> >>> > > > > cycles to get in and solve the problem.  the
> > > shutdown
> > > > > of that
> > > > >     >> >>> > > > > is
> > > > >     >> >>> > > imminent
> > > > >     >> >>> > > > > right now, so we need to solve it asap.
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > *Will STEVENS*
> > > > >     >> >>> > > > > Lead Developer
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > >     >> >>> > paul.angus@shapeblue.com
> > > > >     >> >>> > > >
> > > > >     >> >>> > > > > wrote:
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > > Hi Nathan,
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > Ideally, if you put the template location in
> (or
> > > use
> > > > a
> > > > >     >> >>> > > > > > template
> > > > >     >> >>> > > defined
> > > > >     >> >>> > > > > > in)  test_data.py then the actual location can
> > be
> > > > >     >> >>> > > > > > overridden by
> > > > >     >> >>> > > anyone
> > > > >     >> >>> > > > > > testing.
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > For Trillian, we've copied all of the
> templates
> > > that
> > > > > people
> > > > >     >> >>> > > > > > have
> > > > >     >> >>> > > define
> > > > >     >> >>> > > > > to
> > > > >     >> >>> > > > > > a local repo and then replace the URLs in
> > > > > test_data.py to
> > > > >     >> >>> > > > > > reduce
> > > > >     >> >>> > > > > bandwidth
> > > > >     >> >>> > > > > > use and download times.
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > Ie:
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >             "bootableIso":
> > > > >     >> >>> > > > > >                 {
> > > > >     >> >>> > > > > >                     "displaytext": "Test
> > Bootable
> > > > > ISO",
> > > > >     >> >>> > > > > >                     "name": "testISO",
> > > > >     >> >>> > > > > >                     "bootable": True,
> > > > >     >> >>> > > > > >                     "ispublic": False,
> > > > >     >> >>> > > > > >                     "url": "{{
> > > > marvin_images_location
> > > > >     >> >>> > > > > > }}/TinyCore-current.iso",
> > > > >     >> >>> > > > > >                     "ostype": 'Other Linux
> > > > (64-bit)',
> > > > >     >> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > >     >> >>> > > > > >         },
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > I thought that we had come up with a solution
> > for
> > > > >     >> >>> > download.cloud.com
> > > > >     >> >>> > > ,
> > > > >     >> >>> > > > by
> > > > >     >> >>> > > > > > having a mirrorlist hosted in Community Apache
> > > > > 'space' with
> > > > >     >> >>> anyone
> > > > >     >> >>> > > able
> > > > >     >> >>> > > > > to
> > > > >     >> >>> > > > > > out themselves forward as a mirror.
> > > > >     >> >>> > > > > > But I must admit I lost track of whether
> anyone
> > > made
> > > > > the
> > > > >     >> >>> requisite
> > > > >     >> >>> > > > > changes
> > > > >     >> >>> > > > > > in code....
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > Kind regards,
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > Paul Angus
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > paul.angus@shapeblue.com
> > > > >     >> >>> > > > > > www.shapeblue.com
> > > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N
> > > 4HSUK
> > > > >     >> >>> > > > > > @shapeblue
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > -----Original Message-----
> > > > >     >> >>> > > > > > From: williamstevens@gmail.com
> > > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
> > > > >     >> >>> > On
> > > > >     >> >>> > > > > > Behalf Of Will Stevens
> > > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
> > > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
> > > > >     >> >>> > > > > > Subject: Re: Modern template hosting
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > this is a hard questions.  in general, we
> should
> > > be
> > > > > setting
> > > > >     >> >>> > > > > > up a
> > > > >     >> >>> > > mirror
> > > > >     >> >>> > > > > on
> > > > >     >> >>> > > > > > some cloudstack/apache domain and then mirror
> to
> > > > other
> > > > >     >> >>> > > > > > provided
> > > > >     >> >>> > > > > templates.
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > we MUST come up with a solution to deprecate
> > > > >     >> >>> > > > > > 'download.cloud.com
> > > > >     >> >>> ',
> > > > >     >> >>> > > > that
> > > > >     >> >>> > > > > > is going to be going away any day now.
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > i don't know the right way to solve this to be
> > > > > honest, but
> > > > >     >> >>> > > > > > if you
> > > > >     >> >>> > > have
> > > > >     >> >>> > > > > > ideas, i am willing to help.
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > *Will STEVENS*
> > > > >     >> >>> > > > > > Lead Developer
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan
> > Johnson <
> > > > >     >> >>> njohnson@ena.com
> > > > >     >> >>> > >
> > > > >     >> >>> > > > > wrote:
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > > > > So not to re-open a can of worms, but I’m
> in a
> > > > > situation
> > > > >     >> >>> > > > > > > where
> > > > >     >> >>> I
> > > > >     >> >>> > > need
> > > > >     >> >>> > > > > > > to come up with a Marvin component test that
> > > > > depends on a
> > > > >     >> >>> > template
> > > > >     >> >>> > > > > > > based on a kernel that’s relatively new,
> i.e.,
> > > > > newer than
> > > > >     >> >>> Centos
> > > > >     >> >>> > > 5.3
> > > > >     >> >>> > > > /
> > > > >     >> >>> > > > > > Ubuntu 10.04 .
> > > > >     >> >>> > > > > > > I see openvm.eu has a suitable template
> > (Ubuntu
> > > > > 16.0.4
> > > > >     >> >>> > > > > > > for
> > > > >     >> >>> KVM),
> > > > >     >> >>> > > but
> > > > >     >> >>> > > > > > > from looking at the thread "Migrating
> > CloudStack
> > > > > content
> > > > >     >> >>> > > > > > > from download.cloud.com” it looks like
> there
> > is
> > > > >     >> >>> > > > > > > resistance to using
> > > > >     >> >>> > > this
> > > > >     >> >>> > > > at
> > > > >     >> >>> > > > > > > least for hosting system vm templates over
> > > > concerns
> > > > > of
> > > > >     >> >>> > neutrality.
> > > > >     >> >>> > > > > > > Would this be suitable for a component test?
> > If
> > > > > not,
> > > > >     >> >>> > > > > > > what is a
> > > > >     >> >>> > > > > “blessed”
> > > > >     >> >>> > > > > > > template location?
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > > Thanks in advance!
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > > Nathan Johnson
> > > > >     >> >>> > > > > > > R&D Engineer
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12
> Nashville,
> > TN
> > > > > 37211
> > > > >     >> >>> > > > > > > General Office: 615-312-6000
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > > website | blog | support
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > > >
> > > > >     >> >>> > > > > >
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > > > DISCLAIMER
> > > > >     >> >>> > > > > ==========
> > > > >     >> >>> > > > > This e-mail may contain privileged and
> > confidential
> > > > >     >> >>> > > > > information
> > > > >     >> >>> which
> > > > >     >> >>> > > is
> > > > >     >> >>> > > > > the property of Accelerite, a Persistent Systems
> > > > > business. It
> > > > >     >> >>> > > > > is
> > > > >     >> >>> > > intended
> > > > >     >> >>> > > > > only for the use of the individual or entity to
> > > which
> > > > > it is
> > > > >     >> >>> > addressed.
> > > > >     >> >>> > > If
> > > > >     >> >>> > > > > you are not the intended recipient, you are not
> > > > > authorized to
> > > > >     >> >>> > > > > read,
> > > > >     >> >>> > > > retain,
> > > > >     >> >>> > > > > copy, print, distribute or use this message. If
> > you
> > > > have
> > > > >     >> >>> > > > > received
> > > > >     >> >>> > this
> > > > >     >> >>> > > > > communication in error, please notify the sender
> > and
> > > > > delete
> > > > >     >> >>> > > > > all
> > > > >     >> >>> > copies
> > > > >     >> >>> > > of
> > > > >     >> >>> > > > > this message. Accelerite, a Persistent Systems
> > > > business
> > > > > does
> > > > >     >> >>> > > > > not
> > > > >     >> >>> > accept
> > > > >     >> >>> > > > any
> > > > >     >> >>> > > > > liability for virus infected mails.
> > > > >     >> >>> > > > >
> > > > >     >> >>> > > >
> > > > >     >> >>> > >
> > > > >     >> >>> >
> > > > >     >> >>>
> > > > >     >> >
> > > > >     >> >abhinandan.prateek@shapeblue.com
> > > > >     >> >www.shapeblue.com
> > > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> > > > >     >> >
> > > > >     >> >
> > > > >     >> >
> > > > >     >>
> > > > >     >> abhinandan.prateek@shapeblue.com
> > > > >     >> www.shapeblue.com
> > > > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > >     >> @shapeblue
> > > > >     >>
> > > > >     >>
> > > > >     >>
> > > > >     >>
> > > > >     >
> > > > >
> > > > >
> > > > >
> > > > > daan.hoogland@shapeblue.com
> > > > > www.shapeblue.com
> > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > @shapeblue
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Rafael Weingärtner
> > > >
> > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> > >
> >
>
>
>
> --
> Rafael Weingärtner
>

Re: Modern template hosting

Posted by Rafael Weingärtner <ra...@gmail.com>.

Yes it makes sense. I agree with you regarding the legacy application. I
forgot them, my bad. Considering legacy application, I do agree that using
the HTTP 300 method will not work.

Now regarding the second point in the discussion; I think clients have a
better position to decide, which mirror is the best for them. The same
process you do to check the mirror latency, or check if the mirror is
online could be executed on the client side. As I said before, this shifts
the responsibility to the client side (this may be a good or a bad thing
depending on the perspective).

Anyways, I do not think that it would be feasible to use the approach I
proposed given legacy systems. When I proposed, I did not think of this
problem.

Besides that comment, I think the system is ok ;)

Before, you also asked opinions about the use of “cloudstack-www” as a repo
for this mirror list file. I am in favor of that. Then, it would not
require setting up other repositories and routines to get data from Github
to a web server at Apache.

On Fri, Mar 3, 2017 at 11:13 AM, Will Stevens <ws...@cloudops.com> wrote:

> So the main issue I see with this is the following.
>
> This implementation is designed to target legacy installations which will
> be affected when download.cloud.com disappears.  These people will need to
> make a DB change to replace the 'download.cloud.com' with some other url
> (maybe 'download.cloudstack.org').  Once the DB has been updated, we can
> not expect anything else of the client, they need to be able to just
> continue operation as they were without any need to rebuild or upgrade
> their ACS.
>
> If we try to force the decision to the client, then we break the legacy
> implementations.  Since the SSVM is likely going to be the client in some
> cases, and since it does not already have the logic to handle the 300
> approach correctly, I feel like it is not a viable solution for the legacy
> deployments.
>
> In addition to that.  I don't think the client is in any better position to
> make the mirror decision than I am.  I am likely able to make a more
> 'educated' decision than the client would be able to because I can do
> different tests on the endpoint(s) before making a final decision.
>
> Does this make sense or am I missing something obvious?
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
> rafaelweingartner@gmail.com> wrote:
>
> > I think I understood you, but I did not understand you.
> >
> > Let me see if I can create an example to illustrate how we could do using
> > the HTTP 300 code.
> >
> > The HTTP 300 code indicates clients that he/she/it has multiple choices
> to
> > access the requested resource. If the request is a HEAD, this is the only
> > thing that the server returns. If the request if a GET, then the server
> > returns the choices for the client. The response type is defined on the
> > server based on the Content-Type, user agent and others.
> >
> > Let’s say the server receives a GET request and informs the content-type
> as
> > JSON (application/JSON). Your application could get the mirrors list, do
> > the appending process for the requested resource path, and then return a
> > JSON array with the possible mirror the client can use. Then, the client
> > gets this list and does the processing required to select a mirror.
> >
> > Now let’s say that a human access the link using a browser. Then, the
> > server should receive something like (text/plain) as the content-type; I
> > think depending on the browser this field may differ a little (it would
> > require some checking). Anyways, the server detects that it is a “human”
> > requesting the resource, then we could serve an HTML page with a
> Javascript
> > that uses the mirror list. This Javascript could do some checking and
> > choose the best mirror for that specific.
> >
> > Why do I think this approach is interesting?
> > The application you developed would be used only to retrieve valid and
> > trusted sources of system VM images (a trusted repository for mirrors
> that
> > we as a community vouch for).  Also, we shift the decision process
> > regarding mirrors from the server to the client. Then, it is up to
> clients
> > to select mirrors, and not up to us (or some of our implementations).
> >
> > Did this help?
> > BTW: I am not saying I am against the way you proposed, which would work
> > fine. It is merely a suggestion using a different perspective.
> >
> > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <wi...@gmail.com>
> > wrote:
> >
> > > @rafael: in general, I don't think the client should ever be making a
> > > choice about a mirror. We have to assume we are working with a scripted
> > > application and anything hard coded in that implementation on the
> client
> > > side is a risk.
> > >
> > > I may not be understanding the use of the 300 approach, so I will
> > research
> > > it to see if I can make it fit.
> > >
> > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
> > rafaelweingartner@gmail.com>
> > > wrote:
> > >
> > > Will great job.
> > >
> > > I had the same doubt as Daan.
> > >
> > > Have you considered using HTTP 300 working mode? Then, we could let the
> > > client decide which mirror is the best(closest?). Section “10.3.1” of
> [1]
> > > talks about it; this would put the pressure on deciding upon a mirror
> on
> > > the client side, but I think it would be fair.
> > >
> > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
> > >
> > > PS: You really like very short variables names!
> > >
> > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
> > daan.hoogland@shapeblue.com
> > > >
> > > wrote:
> > >
> > > > Nice little thing Will,
> > > >
> > > > One question: if I read the code correctly it ‘go’es and tries all
> > > mirrors
> > > > at once and whichever responses first is redirected to the client.
> This
> > > > might well be the same every time. This might be the one closest to
> the
> > > > mirror302 and not to the client. Is that correct or did I miss a
> > weighing
> > > > algorithm hidden in there?
> > > >
> > > > Good coding,
> > > >
> > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
> > > >
> > > >     Hey All,
> > > >     Please review this repo and tell me what you think:
> > > >     https://github.com/swill/mirror302
> > > >
> > > >     Let me know if you have questions or you would like me to make
> > > changes
> > > >     before I bring this topic up with ASF Infra and open a ticket to
> > get
> > > > this
> > > >     implemented.
> > > >
> > > >     Cheers,
> > > >
> > > >     *Will Stevens*
> > > >
> > > >
> > > >
> > > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> > wstevens@cloudops.com
> > > >
> > > > wrote:
> > > >
> > > >     > I am building a short term solution right now.  I hopefully
> will
> > > > have the
> > > >     > building blocks in place by the weekend so we can start working
> > > with
> > > > Infra
> > > >     > to get it in place.
> > > >     >
> > > >     > We will have a web server which we can point something like
> > > >     > downloads.cloudstack.org (or whatever url) at.  Then we will
> > have
> > > > an ASF
> > > >     > repo which tracks a mirror list and exposes a static site to be
> > > able
> > > > to
> > > >     > query the mirror list.
> > > >     >
> > > >     > The web server will receive a request and will do a 302
> redirect
> > to
> > > > the
> > > >     > appropriate resource on one of the mirrors.
> > > >     >
> > > >     > This gets us started.  Legacy environments will have to do a DB
> > > > change to
> > > >     > change from 'download.cloud.com' to 'downloads.cloudstack.org'
> > (or
> > > >     > whatever), but otherwise it should be pretty simple.
> > > >     >
> > > >     > We can improve how we deliver templates going forward, but this
> > > > obviously
> > > >     > requires some discussion still.  I will try to buy us some time
> > > with
> > > > an
> > > >     > implementation which solves our problems today...
> > > >     >
> > > >     > *Will STEVENS*
> > > >     > Lead Developer
> > > >     >
> > > >     > <https://goo.gl/NYZ8KK>
> > > >     >
> > > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > > >     > abhinandan.prateek@shapeblue.com> wrote:
> > > >     >
> > > >     >>
> > > >     >> Initial seeding is a manual step and that is not going to
> change
> > > in
> > > > near
> > > >     >> future. A handy list of official places from where these
> > templates
> > > > can be
> > > >     >> downloaded won’t harm this part of the installation. Just
> allow
> > > > anyone
> > > >     >> credible who follows democratic process to publish their
> > templates
> > > > with
> > > >     >> apache’s blessings.
> > > >     >>
> > > >     >>
> > > >     >> Coming to urls hard coded in db, the popular suggestion is to
> > > point
> > > > them
> > > >     >> to mirrors. This looks good but this is going to take some
> > effort
> > > > that has
> > > >     >> not materialised yet. Once this change is made it will be
> going
> > in
> > > > some
> > > >     >> future release like 4.11; the problem still remains for the
> > > > releases that
> > > >     >> are already out there. In case access to download.cloud.com
> is
> > > > dropped
> > > >     >> at some point we will have to document the procedure such that
> > > > people know
> > > >     >> how to make these stable releases work. That is where the
> > > procedure
> > > >     >> documented here https://shankerbalan.net/blog/
> > > >     >> seed-cloudstack-templates-offline/ should be officially
> adapted
> > > > with a
> > > >     >> marketplace to pick up templates of choosing. Maybe we can
> have
> > a
> > > > pre-setup
> > > >     >> script to make choices instead of written wiki. Note that this
> > > will
> > > > be
> > > >     >> anyway needed by the older releases.
> > > >     >>
> > > >     >>
> > > >     >>
> > > >     >>
> > > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <pa...@shapeblue.com>
> > > > wrote:
> > > >     >>
> > > >     >> >-1
> > > >     >> >
> > > >     >> >We are trying to get MORE people to use CloudStack.  And
> > raising
> > > > the
> > > >     >> barrier to entry does nothing to help that.
> > > >     >> >
> > > >     >> >
> > > >     >> >Kind regards,
> > > >     >> >
> > > >     >> >Paul Angus
> > > >     >> >
> > > >     >> >paul.angus@shapeblue.com
> > > >     >> >www.shapeblue.com
> > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > >     >> >@shapeblue
> > > >     >> >
> > > >     >> >
> > > >     >> >
> > > >     >> >
> > > >     >> >-----Original Message-----
> > > >     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@
> > > shapeblue.com]
> > > >     >> >Sent: 01 March 2017 09:39
> > > >     >> >To: dev@cloudstack.apache.org
> > > >     >> >Subject: Re: Modern template hosting
> > > >     >> >
> > > >     >> >+1
> > > >     >> >
> > > >     >> >After seeing this discussion running in circles several
> times,
> > I
> > > > think
> > > >     >> we should at least get started with the simplest option.
> > > >     >> >
> > > >     >> >
> > > >     >> >
> > > >     >> >
> > > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
> > chiradeepv@gmail.com>
> > > > wrote:
> > > >     >> >
> > > >     >> >>The alternative foolproof way is to simply not provide the
> > > > automatic
> > > >     >> >>download.
> > > >     >> >>
> > > >     >> >>Just document the mirror list URL where the templates can be
> > > > found. I
> > > >     >> >>don't know why people reject this option.
> > > >     >> >>
> > > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
> > > templates-offline/
> > > >     >> >>
> > > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> > > >     >> >><wi...@gmail.com>
> > > >     >> >>wrote:
> > > >     >> >>
> > > >     >> >>> OK. Thanks for the heads up.
> > > >     >> >>>
> > > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > chiradeepv@gmail.com>
> > > >     >> wrote:
> > > >     >> >>>
> > > >     >> >>> > Sounds workable. The downloader code in the SSVM won't
> > > follow
> > > >     >> >>> > redirects I think.
> > > >     >> >>> > https://github.com/apache/cloudstack/blob/
> > > 5511065fc20787619d
> > > > 9cd0444
> > > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
> > > >     >> >>> > HttpTemplateDownloader.java#L93
> > > >     >> >>> > https://goo.gl/dSi0r5
> > > >     >> >>> >  Might need to add
> > > >     >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > >     >> >>> >
> > > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> > > >     >> >>> > <ws...@cloudops.com>
> > > >     >> >>> > wrote:
> > > >     >> >>> >
> > > >     >> >>> > > We haven't opened a ticket yet because we don't have a
> > > > strategy
> > > >     >> yet.
> > > >     >> >>> > >
> > > >     >> >>> > > What do you guys think of this:
> > > >     >> >>> > > - We setup a new github repo in the 'apache' org which
> > > > consists
> > > >     >> >>> > > of a
> > > >     >> >>> > single
> > > >     >> >>> > > file with a list of active/supported mirrors.
> > > >     >> >>> > > - I write a small web server, distributed as a binary,
> > > > which can
> > > >     >> >>> > > be
> > > >     >> >>> > hosted
> > > >     >> >>> > > by ASF Infra.  This web server will query the current
> > list
> > > > of
> > > >     >> >>> > > mirrors
> > > >     >> >>> and
> > > >     >> >>> > > will select one and then do a 302 redirect to that
> > mirror.
> > > >     >> >>> > >
> > > >     >> >>> > > The act of 'choosing' a mirror could be done in a
> number
> > > of
> > > > ways.
> > > >     >> >>> > > - If we want to define an order, then it could just
> try
> > > > from the
> > > >     >> >>> > > top of
> > > >     >> >>> > the
> > > >     >> >>> > > list and work its way down.  It would curl the target
> to
> > > > make
> > > >     >> >>> > > sure it
> > > >     >> >>> > gets
> > > >     >> >>> > > a 200 and if it does, it would do a 302 redirect.
> > > >     >> >>> > > - Or, if we want to distribute the load across the
> > > mirrors,
> > > > we
> > > >     >> >>> > > could
> > > >     >> >>> pick
> > > >     >> >>> > > from the list randomly.  Again, doing a curl to verify
> > the
> > > > mirror
> > > >     >> >>> > > is up
> > > >     >> >>> > and
> > > >     >> >>> > > then doing a redirect.
> > > >     >> >>> > > - If we want to get fancy, we could do a reverse IP
> > lookup
> > > > and
> > > >     >> >>> > > try to
> > > >     >> >>> > match
> > > >     >> >>> > > the requester with their closest geographical mirror.
> > > >     >> >>> > >
> > > >     >> >>> > > Thoughts?
> > > >     >> >>> > >
> > > >     >> >>> > > *Will STEVENS*
> > > >     >> >>> > > Lead Developer
> > > >     >> >>> > >
> > > >     >> >>> > > <https://goo.gl/NYZ8KK>
> > > >     >> >>> > >
> > > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > >     >> >>> chiradeepv@gmail.com
> > > >     >> >>> > >
> > > >     >> >>> > > wrote:
> > > >     >> >>> > >
> > > >     >> >>> > > > What steps are needed to set up a mirror? What does
> > > Infra
> > > > need
> > > >     >> to do?
> > > >     >> >>> > Has
> > > >     >> >>> > > > anybody filed a ticket with Infra?
> > > >     >> >>> > > >
> > > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > >     >> >>> > > > raja.pullela@accelerite.com>
> > > >     >> >>> > > > wrote:
> > > >     >> >>> > > >
> > > >     >> >>> > > > > Hi will,
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > I believe, we didn’t get to close ‘getting a
> mirror
> > on
> > > > Apache’
> > > >     >> >>> > because
> > > >     >> >>> > > we
> > > >     >> >>> > > > > needed someone on the Apache Infra side to close
> > this.
> > > > BTW,
> > > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages this?)
> > has
> > > >     >> >>> > > > > all/most of
> > > >     >> >>> > the
> > > >     >> >>> > > > > content.  Once we can close on the Apache mirror
> for
> > > > hosting
> > > >     >> >>> > > > > the
> > > >     >> >>> > > > content, I
> > > >     >> >>> > > > > can help assist getting the content there.
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > For now, we have replicated the
> download.cloud.com
> > > > content to
> > > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
> > > >     >> >>> > > > > Also, we are working on a set of steps/procedure
> to
> > > > help with
> > > >     >> >>> > > > > this change.  I will update everyone in about a
> > week’s
> > > > time
> > > >     >> >>> > > > > on the
> > > >     >> >>> > details.
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > Best,
> > > >     >> >>> > > > > Raja Pullela
> > > >     >> >>> > > > > Engineering Team,
> > > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA,
> > > 95054
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com
> on
> > > > behalf of
> > > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com on behalf
> > of
> > > >     >> >>> > wstevens@cloudops.com>
> > > >     >> >>> > > > > wrote:
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > unfortunately the template mirror conversation got
> > > > caught up
> > > >     >> >>> > > > > in
> > > >     >> >>> > details
> > > >     >> >>> > > > and
> > > >     >> >>> > > > > nobody took the lead on implementing a solution.
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > citrix has been pinging me every couple months to
> > say
> > > > 'dude,
> > > >     >> >>> > > > > we
> > > >     >> >>> need
> > > >     >> >>> > to
> > > >     >> >>> > > > > remove the dependency on download.citrix.com',
> but
> > i
> > > > have not
> > > >     >> >>> > > > > had
> > > >     >> >>> > the
> > > >     >> >>> > > > > cycles to get in and solve the problem.  the
> > shutdown
> > > > of that
> > > >     >> >>> > > > > is
> > > >     >> >>> > > imminent
> > > >     >> >>> > > > > right now, so we need to solve it asap.
> > > >     >> >>> > > > >
> > > >     >> >>> > > > >
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > *Will STEVENS*
> > > >     >> >>> > > > > Lead Developer
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > >     >> >>> > paul.angus@shapeblue.com
> > > >     >> >>> > > >
> > > >     >> >>> > > > > wrote:
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > > Hi Nathan,
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > Ideally, if you put the template location in (or
> > use
> > > a
> > > >     >> >>> > > > > > template
> > > >     >> >>> > > defined
> > > >     >> >>> > > > > > in)  test_data.py then the actual location can
> be
> > > >     >> >>> > > > > > overridden by
> > > >     >> >>> > > anyone
> > > >     >> >>> > > > > > testing.
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > For Trillian, we've copied all of the templates
> > that
> > > > people
> > > >     >> >>> > > > > > have
> > > >     >> >>> > > define
> > > >     >> >>> > > > > to
> > > >     >> >>> > > > > > a local repo and then replace the URLs in
> > > > test_data.py to
> > > >     >> >>> > > > > > reduce
> > > >     >> >>> > > > > bandwidth
> > > >     >> >>> > > > > > use and download times.
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > Ie:
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >             "bootableIso":
> > > >     >> >>> > > > > >                 {
> > > >     >> >>> > > > > >                     "displaytext": "Test
> Bootable
> > > > ISO",
> > > >     >> >>> > > > > >                     "name": "testISO",
> > > >     >> >>> > > > > >                     "bootable": True,
> > > >     >> >>> > > > > >                     "ispublic": False,
> > > >     >> >>> > > > > >                     "url": "{{
> > > marvin_images_location
> > > >     >> >>> > > > > > }}/TinyCore-current.iso",
> > > >     >> >>> > > > > >                     "ostype": 'Other Linux
> > > (64-bit)',
> > > >     >> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > >     >> >>> > > > > >         },
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > I thought that we had come up with a solution
> for
> > > >     >> >>> > download.cloud.com
> > > >     >> >>> > > ,
> > > >     >> >>> > > > by
> > > >     >> >>> > > > > > having a mirrorlist hosted in Community Apache
> > > > 'space' with
> > > >     >> >>> anyone
> > > >     >> >>> > > able
> > > >     >> >>> > > > > to
> > > >     >> >>> > > > > > out themselves forward as a mirror.
> > > >     >> >>> > > > > > But I must admit I lost track of whether anyone
> > made
> > > > the
> > > >     >> >>> requisite
> > > >     >> >>> > > > > changes
> > > >     >> >>> > > > > > in code....
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > Kind regards,
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > Paul Angus
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > paul.angus@shapeblue.com
> > > >     >> >>> > > > > > www.shapeblue.com
> > > >     >> >>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N
> > 4HSUK
> > > >     >> >>> > > > > > @shapeblue
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > -----Original Message-----
> > > >     >> >>> > > > > > From: williamstevens@gmail.com
> > > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
> > > >     >> >>> > On
> > > >     >> >>> > > > > > Behalf Of Will Stevens
> > > >     >> >>> > > > > > Sent: 24 February 2017 16:30
> > > >     >> >>> > > > > > To: dev@cloudstack.apache.org
> > > >     >> >>> > > > > > Subject: Re: Modern template hosting
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > this is a hard questions.  in general, we should
> > be
> > > > setting
> > > >     >> >>> > > > > > up a
> > > >     >> >>> > > mirror
> > > >     >> >>> > > > > on
> > > >     >> >>> > > > > > some cloudstack/apache domain and then mirror to
> > > other
> > > >     >> >>> > > > > > provided
> > > >     >> >>> > > > > templates.
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > we MUST come up with a solution to deprecate
> > > >     >> >>> > > > > > 'download.cloud.com
> > > >     >> >>> ',
> > > >     >> >>> > > > that
> > > >     >> >>> > > > > > is going to be going away any day now.
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > i don't know the right way to solve this to be
> > > > honest, but
> > > >     >> >>> > > > > > if you
> > > >     >> >>> > > have
> > > >     >> >>> > > > > > ideas, i am willing to help.
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > *Will STEVENS*
> > > >     >> >>> > > > > > Lead Developer
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan
> Johnson <
> > > >     >> >>> njohnson@ena.com
> > > >     >> >>> > >
> > > >     >> >>> > > > > wrote:
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > > > > So not to re-open a can of worms, but I’m in a
> > > > situation
> > > >     >> >>> > > > > > > where
> > > >     >> >>> I
> > > >     >> >>> > > need
> > > >     >> >>> > > > > > > to come up with a Marvin component test that
> > > > depends on a
> > > >     >> >>> > template
> > > >     >> >>> > > > > > > based on a kernel that’s relatively new, i.e.,
> > > > newer than
> > > >     >> >>> Centos
> > > >     >> >>> > > 5.3
> > > >     >> >>> > > > /
> > > >     >> >>> > > > > > Ubuntu 10.04 .
> > > >     >> >>> > > > > > > I see openvm.eu has a suitable template
> (Ubuntu
> > > > 16.0.4
> > > >     >> >>> > > > > > > for
> > > >     >> >>> KVM),
> > > >     >> >>> > > but
> > > >     >> >>> > > > > > > from looking at the thread "Migrating
> CloudStack
> > > > content
> > > >     >> >>> > > > > > > from download.cloud.com” it looks like there
> is
> > > >     >> >>> > > > > > > resistance to using
> > > >     >> >>> > > this
> > > >     >> >>> > > > at
> > > >     >> >>> > > > > > > least for hosting system vm templates over
> > > concerns
> > > > of
> > > >     >> >>> > neutrality.
> > > >     >> >>> > > > > > > Would this be suitable for a component test?
> If
> > > > not,
> > > >     >> >>> > > > > > > what is a
> > > >     >> >>> > > > > “blessed”
> > > >     >> >>> > > > > > > template location?
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > > Thanks in advance!
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > > Nathan Johnson
> > > >     >> >>> > > > > > > R&D Engineer
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville,
> TN
> > > > 37211
> > > >     >> >>> > > > > > > General Office: 615-312-6000
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > > website | blog | support
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > > >
> > > >     >> >>> > > > > >
> > > >     >> >>> > > > >
> > > >     >> >>> > > > >
> > > >     >> >>> > > > >
> > > >     >> >>> > > > >
> > > >     >> >>> > > > >
> > > >     >> >>> > > > >
> > > >     >> >>> > > > > DISCLAIMER
> > > >     >> >>> > > > > ==========
> > > >     >> >>> > > > > This e-mail may contain privileged and
> confidential
> > > >     >> >>> > > > > information
> > > >     >> >>> which
> > > >     >> >>> > > is
> > > >     >> >>> > > > > the property of Accelerite, a Persistent Systems
> > > > business. It
> > > >     >> >>> > > > > is
> > > >     >> >>> > > intended
> > > >     >> >>> > > > > only for the use of the individual or entity to
> > which
> > > > it is
> > > >     >> >>> > addressed.
> > > >     >> >>> > > If
> > > >     >> >>> > > > > you are not the intended recipient, you are not
> > > > authorized to
> > > >     >> >>> > > > > read,
> > > >     >> >>> > > > retain,
> > > >     >> >>> > > > > copy, print, distribute or use this message. If
> you
> > > have
> > > >     >> >>> > > > > received
> > > >     >> >>> > this
> > > >     >> >>> > > > > communication in error, please notify the sender
> and
> > > > delete
> > > >     >> >>> > > > > all
> > > >     >> >>> > copies
> > > >     >> >>> > > of
> > > >     >> >>> > > > > this message. Accelerite, a Persistent Systems
> > > business
> > > > does
> > > >     >> >>> > > > > not
> > > >     >> >>> > accept
> > > >     >> >>> > > > any
> > > >     >> >>> > > > > liability for virus infected mails.
> > > >     >> >>> > > > >
> > > >     >> >>> > > >
> > > >     >> >>> > >
> > > >     >> >>> >
> > > >     >> >>>
> > > >     >> >
> > > >     >> >abhinandan.prateek@shapeblue.com
> > > >     >> >www.shapeblue.com
> > > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
> > > >     >> >
> > > >     >> >
> > > >     >> >
> > > >     >>
> > > >     >> abhinandan.prateek@shapeblue.com
> > > >     >> www.shapeblue.com
> > > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > >     >> @shapeblue
> > > >     >>
> > > >     >>
> > > >     >>
> > > >     >>
> > > >     >
> > > >
> > > >
> > > >
> > > > daan.hoogland@shapeblue.com
> > > > www.shapeblue.com
> > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > @shapeblue
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> > >
> >
> >
> >
> > --
> > Rafael Weingärtner
> >
>



-- 
Rafael Weingärtner

Re: Modern template hosting

Posted by Will Stevens <ws...@cloudops.com>.

So the main issue I see with this is the following.

This implementation is designed to target legacy installations which will
be affected when download.cloud.com disappears.  These people will need to
make a DB change to replace the 'download.cloud.com' with some other url
(maybe 'download.cloudstack.org').  Once the DB has been updated, we can
not expect anything else of the client, they need to be able to just
continue operation as they were without any need to rebuild or upgrade
their ACS.

If we try to force the decision to the client, then we break the legacy
implementations.  Since the SSVM is likely going to be the client in some
cases, and since it does not already have the logic to handle the 300
approach correctly, I feel like it is not a viable solution for the legacy
deployments.

In addition to that.  I don't think the client is in any better position to
make the mirror decision than I am.  I am likely able to make a more
'educated' decision than the client would be able to because I can do
different tests on the endpoint(s) before making a final decision.

Does this make sense or am I missing something obvious?

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner <
rafaelweingartner@gmail.com> wrote:

> I think I understood you, but I did not understand you.
>
> Let me see if I can create an example to illustrate how we could do using
> the HTTP 300 code.
>
> The HTTP 300 code indicates clients that he/she/it has multiple choices to
> access the requested resource. If the request is a HEAD, this is the only
> thing that the server returns. If the request if a GET, then the server
> returns the choices for the client. The response type is defined on the
> server based on the Content-Type, user agent and others.
>
> Let’s say the server receives a GET request and informs the content-type as
> JSON (application/JSON). Your application could get the mirrors list, do
> the appending process for the requested resource path, and then return a
> JSON array with the possible mirror the client can use. Then, the client
> gets this list and does the processing required to select a mirror.
>
> Now let’s say that a human access the link using a browser. Then, the
> server should receive something like (text/plain) as the content-type; I
> think depending on the browser this field may differ a little (it would
> require some checking). Anyways, the server detects that it is a “human”
> requesting the resource, then we could serve an HTML page with a Javascript
> that uses the mirror list. This Javascript could do some checking and
> choose the best mirror for that specific.
>
> Why do I think this approach is interesting?
> The application you developed would be used only to retrieve valid and
> trusted sources of system VM images (a trusted repository for mirrors that
> we as a community vouch for).  Also, we shift the decision process
> regarding mirrors from the server to the client. Then, it is up to clients
> to select mirrors, and not up to us (or some of our implementations).
>
> Did this help?
> BTW: I am not saying I am against the way you proposed, which would work
> fine. It is merely a suggestion using a different perspective.
>
> On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <wi...@gmail.com>
> wrote:
>
> > @rafael: in general, I don't think the client should ever be making a
> > choice about a mirror. We have to assume we are working with a scripted
> > application and anything hard coded in that implementation on the client
> > side is a risk.
> >
> > I may not be understanding the use of the 300 approach, so I will
> research
> > it to see if I can make it fit.
> >
> > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
> rafaelweingartner@gmail.com>
> > wrote:
> >
> > Will great job.
> >
> > I had the same doubt as Daan.
> >
> > Have you considered using HTTP 300 working mode? Then, we could let the
> > client decide which mirror is the best(closest?). Section “10.3.1” of [1]
> > talks about it; this would put the pressure on deciding upon a mirror on
> > the client side, but I think it would be fair.
> >
> > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
> >
> > PS: You really like very short variables names!
> >
> > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
> daan.hoogland@shapeblue.com
> > >
> > wrote:
> >
> > > Nice little thing Will,
> > >
> > > One question: if I read the code correctly it ‘go’es and tries all
> > mirrors
> > > at once and whichever responses first is redirected to the client. This
> > > might well be the same every time. This might be the one closest to the
> > > mirror302 and not to the client. Is that correct or did I miss a
> weighing
> > > algorithm hidden in there?
> > >
> > > Good coding,
> > >
> > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
> > >
> > >     Hey All,
> > >     Please review this repo and tell me what you think:
> > >     https://github.com/swill/mirror302
> > >
> > >     Let me know if you have questions or you would like me to make
> > changes
> > >     before I bring this topic up with ASF Infra and open a ticket to
> get
> > > this
> > >     implemented.
> > >
> > >     Cheers,
> > >
> > >     *Will Stevens*
> > >
> > >
> > >
> > >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> wstevens@cloudops.com
> > >
> > > wrote:
> > >
> > >     > I am building a short term solution right now.  I hopefully will
> > > have the
> > >     > building blocks in place by the weekend so we can start working
> > with
> > > Infra
> > >     > to get it in place.
> > >     >
> > >     > We will have a web server which we can point something like
> > >     > downloads.cloudstack.org (or whatever url) at.  Then we will
> have
> > > an ASF
> > >     > repo which tracks a mirror list and exposes a static site to be
> > able
> > > to
> > >     > query the mirror list.
> > >     >
> > >     > The web server will receive a request and will do a 302 redirect
> to
> > > the
> > >     > appropriate resource on one of the mirrors.
> > >     >
> > >     > This gets us started.  Legacy environments will have to do a DB
> > > change to
> > >     > change from 'download.cloud.com' to 'downloads.cloudstack.org'
> (or
> > >     > whatever), but otherwise it should be pretty simple.
> > >     >
> > >     > We can improve how we deliver templates going forward, but this
> > > obviously
> > >     > requires some discussion still.  I will try to buy us some time
> > with
> > > an
> > >     > implementation which solves our problems today...
> > >     >
> > >     > *Will STEVENS*
> > >     > Lead Developer
> > >     >
> > >     > <https://goo.gl/NYZ8KK>
> > >     >
> > >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > >     > abhinandan.prateek@shapeblue.com> wrote:
> > >     >
> > >     >>
> > >     >> Initial seeding is a manual step and that is not going to change
> > in
> > > near
> > >     >> future. A handy list of official places from where these
> templates
> > > can be
> > >     >> downloaded won’t harm this part of the installation. Just allow
> > > anyone
> > >     >> credible who follows democratic process to publish their
> templates
> > > with
> > >     >> apache’s blessings.
> > >     >>
> > >     >>
> > >     >> Coming to urls hard coded in db, the popular suggestion is to
> > point
> > > them
> > >     >> to mirrors. This looks good but this is going to take some
> effort
> > > that has
> > >     >> not materialised yet. Once this change is made it will be going
> in
> > > some
> > >     >> future release like 4.11; the problem still remains for the
> > > releases that
> > >     >> are already out there. In case access to download.cloud.com is
> > > dropped
> > >     >> at some point we will have to document the procedure such that
> > > people know
> > >     >> how to make these stable releases work. That is where the
> > procedure
> > >     >> documented here https://shankerbalan.net/blog/
> > >     >> seed-cloudstack-templates-offline/ should be officially adapted
> > > with a
> > >     >> marketplace to pick up templates of choosing. Maybe we can have
> a
> > > pre-setup
> > >     >> script to make choices instead of written wiki. Note that this
> > will
> > > be
> > >     >> anyway needed by the older releases.
> > >     >>
> > >     >>
> > >     >>
> > >     >>
> > >     >> On 01/03/17, 4:24 PM, "Paul Angus" <pa...@shapeblue.com>
> > > wrote:
> > >     >>
> > >     >> >-1
> > >     >> >
> > >     >> >We are trying to get MORE people to use CloudStack.  And
> raising
> > > the
> > >     >> barrier to entry does nothing to help that.
> > >     >> >
> > >     >> >
> > >     >> >Kind regards,
> > >     >> >
> > >     >> >Paul Angus
> > >     >> >
> > >     >> >paul.angus@shapeblue.com
> > >     >> >www.shapeblue.com
> > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > >     >> >@shapeblue
> > >     >> >
> > >     >> >
> > >     >> >
> > >     >> >
> > >     >> >-----Original Message-----
> > >     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@
> > shapeblue.com]
> > >     >> >Sent: 01 March 2017 09:39
> > >     >> >To: dev@cloudstack.apache.org
> > >     >> >Subject: Re: Modern template hosting
> > >     >> >
> > >     >> >+1
> > >     >> >
> > >     >> >After seeing this discussion running in circles several times,
> I
> > > think
> > >     >> we should at least get started with the simplest option.
> > >     >> >
> > >     >> >
> > >     >> >
> > >     >> >
> > >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
> chiradeepv@gmail.com>
> > > wrote:
> > >     >> >
> > >     >> >>The alternative foolproof way is to simply not provide the
> > > automatic
> > >     >> >>download.
> > >     >> >>
> > >     >> >>Just document the mirror list URL where the templates can be
> > > found. I
> > >     >> >>don't know why people reject this option.
> > >     >> >>
> > >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
> > templates-offline/
> > >     >> >>
> > >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> > >     >> >><wi...@gmail.com>
> > >     >> >>wrote:
> > >     >> >>
> > >     >> >>> OK. Thanks for the heads up.
> > >     >> >>>
> > >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > chiradeepv@gmail.com>
> > >     >> wrote:
> > >     >> >>>
> > >     >> >>> > Sounds workable. The downloader code in the SSVM won't
> > follow
> > >     >> >>> > redirects I think.
> > >     >> >>> > https://github.com/apache/cloudstack/blob/
> > 5511065fc20787619d
> > > 9cd0444
> > >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
> > >     >> >>> > HttpTemplateDownloader.java#L93
> > >     >> >>> > https://goo.gl/dSi0r5
> > >     >> >>> >  Might need to add
> > >     >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
> > >     >> >>> >
> > >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> > >     >> >>> > <ws...@cloudops.com>
> > >     >> >>> > wrote:
> > >     >> >>> >
> > >     >> >>> > > We haven't opened a ticket yet because we don't have a
> > > strategy
> > >     >> yet.
> > >     >> >>> > >
> > >     >> >>> > > What do you guys think of this:
> > >     >> >>> > > - We setup a new github repo in the 'apache' org which
> > > consists
> > >     >> >>> > > of a
> > >     >> >>> > single
> > >     >> >>> > > file with a list of active/supported mirrors.
> > >     >> >>> > > - I write a small web server, distributed as a binary,
> > > which can
> > >     >> >>> > > be
> > >     >> >>> > hosted
> > >     >> >>> > > by ASF Infra.  This web server will query the current
> list
> > > of
> > >     >> >>> > > mirrors
> > >     >> >>> and
> > >     >> >>> > > will select one and then do a 302 redirect to that
> mirror.
> > >     >> >>> > >
> > >     >> >>> > > The act of 'choosing' a mirror could be done in a number
> > of
> > > ways.
> > >     >> >>> > > - If we want to define an order, then it could just try
> > > from the
> > >     >> >>> > > top of
> > >     >> >>> > the
> > >     >> >>> > > list and work its way down.  It would curl the target to
> > > make
> > >     >> >>> > > sure it
> > >     >> >>> > gets
> > >     >> >>> > > a 200 and if it does, it would do a 302 redirect.
> > >     >> >>> > > - Or, if we want to distribute the load across the
> > mirrors,
> > > we
> > >     >> >>> > > could
> > >     >> >>> pick
> > >     >> >>> > > from the list randomly.  Again, doing a curl to verify
> the
> > > mirror
> > >     >> >>> > > is up
> > >     >> >>> > and
> > >     >> >>> > > then doing a redirect.
> > >     >> >>> > > - If we want to get fancy, we could do a reverse IP
> lookup
> > > and
> > >     >> >>> > > try to
> > >     >> >>> > match
> > >     >> >>> > > the requester with their closest geographical mirror.
> > >     >> >>> > >
> > >     >> >>> > > Thoughts?
> > >     >> >>> > >
> > >     >> >>> > > *Will STEVENS*
> > >     >> >>> > > Lead Developer
> > >     >> >>> > >
> > >     >> >>> > > <https://goo.gl/NYZ8KK>
> > >     >> >>> > >
> > >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > >     >> >>> chiradeepv@gmail.com
> > >     >> >>> > >
> > >     >> >>> > > wrote:
> > >     >> >>> > >
> > >     >> >>> > > > What steps are needed to set up a mirror? What does
> > Infra
> > > need
> > >     >> to do?
> > >     >> >>> > Has
> > >     >> >>> > > > anybody filed a ticket with Infra?
> > >     >> >>> > > >
> > >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > >     >> >>> > > > raja.pullela@accelerite.com>
> > >     >> >>> > > > wrote:
> > >     >> >>> > > >
> > >     >> >>> > > > > Hi will,
> > >     >> >>> > > > >
> > >     >> >>> > > > > I believe, we didn’t get to close ‘getting a mirror
> on
> > > Apache’
> > >     >> >>> > because
> > >     >> >>> > > we
> > >     >> >>> > > > > needed someone on the Apache Infra side to close
> this.
> > > BTW,
> > >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages this?)
> has
> > >     >> >>> > > > > all/most of
> > >     >> >>> > the
> > >     >> >>> > > > > content.  Once we can close on the Apache mirror for
> > > hosting
> > >     >> >>> > > > > the
> > >     >> >>> > > > content, I
> > >     >> >>> > > > > can help assist getting the content there.
> > >     >> >>> > > > >
> > >     >> >>> > > > > For now, we have replicated the download.cloud.com
> > > content to
> > >     >> >>> > > > > ‘ s3.download.accelerite.com’.
> > >     >> >>> > > > > Also, we are working on a set of steps/procedure to
> > > help with
> > >     >> >>> > > > > this change.  I will update everyone in about a
> week’s
> > > time
> > >     >> >>> > > > > on the
> > >     >> >>> > details.
> > >     >> >>> > > > >
> > >     >> >>> > > > > Best,
> > >     >> >>> > > > > Raja Pullela
> > >     >> >>> > > > > Engineering Team,
> > >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA,
> > 95054
> > >     >> >>> > > > >
> > >     >> >>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on
> > > behalf of
> > >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com on behalf
> of
> > >     >> >>> > wstevens@cloudops.com>
> > >     >> >>> > > > > wrote:
> > >     >> >>> > > > >
> > >     >> >>> > > > > unfortunately the template mirror conversation got
> > > caught up
> > >     >> >>> > > > > in
> > >     >> >>> > details
> > >     >> >>> > > > and
> > >     >> >>> > > > > nobody took the lead on implementing a solution.
> > >     >> >>> > > > >
> > >     >> >>> > > > > citrix has been pinging me every couple months to
> say
> > > 'dude,
> > >     >> >>> > > > > we
> > >     >> >>> need
> > >     >> >>> > to
> > >     >> >>> > > > > remove the dependency on download.citrix.com', but
> i
> > > have not
> > >     >> >>> > > > > had
> > >     >> >>> > the
> > >     >> >>> > > > > cycles to get in and solve the problem.  the
> shutdown
> > > of that
> > >     >> >>> > > > > is
> > >     >> >>> > > imminent
> > >     >> >>> > > > > right now, so we need to solve it asap.
> > >     >> >>> > > > >
> > >     >> >>> > > > >
> > >     >> >>> > > > >
> > >     >> >>> > > > > *Will STEVENS*
> > >     >> >>> > > > > Lead Developer
> > >     >> >>> > > > >
> > >     >> >>> > > > > <https://goo.gl/NYZ8KK>
> > >     >> >>> > > > >
> > >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > >     >> >>> > paul.angus@shapeblue.com
> > >     >> >>> > > >
> > >     >> >>> > > > > wrote:
> > >     >> >>> > > > >
> > >     >> >>> > > > > > Hi Nathan,
> > >     >> >>> > > > > >
> > >     >> >>> > > > > > Ideally, if you put the template location in (or
> use
> > a
> > >     >> >>> > > > > > template
> > >     >> >>> > > defined
> > >     >> >>> > > > > > in)  test_data.py then the actual location can be
> > >     >> >>> > > > > > overridden by
> > >     >> >>> > > anyone
> > >     >> >>> > > > > > testing.
> > >     >> >>> > > > > >
> > >     >> >>> > > > > > For Trillian, we've copied all of the templates
> that
> > > people
> > >     >> >>> > > > > > have
> > >     >> >>> > > define
> > >     >> >>> > > > > to
> > >     >> >>> > > > > > a local repo and then replace the URLs in
> > > test_data.py to
> > >     >> >>> > > > > > reduce
> > >     >> >>> > > > > bandwidth
> > >     >> >>> > > > > > use and download times.
> > >     >> >>> > > > > >
> > >     >> >>> > > > > > Ie:
> > >     >> >>> > > > > >
> > >     >> >>> > > > > >             "bootableIso":
> > >     >> >>> > > > > >                 {
> > >     >> >>> > > > > >                     "displaytext": "Test Bootable
> > > ISO",
> > >     >> >>> > > > > >                     "name": "testISO",
> > >     >> >>> > > > > >                     "bootable": True,
> > >     >> >>> > > > > >                     "ispublic": False,
> > >     >> >>> > > > > >                     "url": "{{
> > marvin_images_location
> > >     >> >>> > > > > > }}/TinyCore-current.iso",
> > >     >> >>> > > > > >                     "ostype": 'Other Linux
> > (64-bit)',
> > >     >> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > >     >> >>> > > > > >         },
> > >     >> >>> > > > > >
> > >     >> >>> > > > > >
> > >     >> >>> > > > > >
> > >     >> >>> > > > > > I thought that we had come up with a solution for
> > >     >> >>> > download.cloud.com
> > >     >> >>> > > ,
> > >     >> >>> > > > by
> > >     >> >>> > > > > > having a mirrorlist hosted in Community Apache
> > > 'space' with
> > >     >> >>> anyone
> > >     >> >>> > > able
> > >     >> >>> > > > > to
> > >     >> >>> > > > > > out themselves forward as a mirror.
> > >     >> >>> > > > > > But I must admit I lost track of whether anyone
> made
> > > the
> > >     >> >>> requisite
> > >     >> >>> > > > > changes
> > >     >> >>> > > > > > in code....
> > >     >> >>> > > > > >
> > >     >> >>> > > > > >
> > >     >> >>> > > > > >
> > >     >> >>> > > > > >
> > >     >> >>> > > > > > Kind regards,
> > >     >> >>> > > > > >
> > >     >> >>> > > > > > Paul Angus
> > >     >> >>> > > > > >
> > >     >> >>> > > > > > paul.angus@shapeblue.com
> > >     >> >>> > > > > > www.shapeblue.com
> > >     >> >>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N
> 4HSUK
> > >     >> >>> > > > > > @shapeblue
> > >     >> >>> > > > > >
> > >     >> >>> > > > > >
> > >     >> >>> > > > > >
> > >     >> >>> > > > > >
> > >     >> >>> > > > > > -----Original Message-----
> > >     >> >>> > > > > > From: williamstevens@gmail.com
> > >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
> > >     >> >>> > On
> > >     >> >>> > > > > > Behalf Of Will Stevens
> > >     >> >>> > > > > > Sent: 24 February 2017 16:30
> > >     >> >>> > > > > > To: dev@cloudstack.apache.org
> > >     >> >>> > > > > > Subject: Re: Modern template hosting
> > >     >> >>> > > > > >
> > >     >> >>> > > > > > this is a hard questions.  in general, we should
> be
> > > setting
> > >     >> >>> > > > > > up a
> > >     >> >>> > > mirror
> > >     >> >>> > > > > on
> > >     >> >>> > > > > > some cloudstack/apache domain and then mirror to
> > other
> > >     >> >>> > > > > > provided
> > >     >> >>> > > > > templates.
> > >     >> >>> > > > > >
> > >     >> >>> > > > > > we MUST come up with a solution to deprecate
> > >     >> >>> > > > > > 'download.cloud.com
> > >     >> >>> ',
> > >     >> >>> > > > that
> > >     >> >>> > > > > > is going to be going away any day now.
> > >     >> >>> > > > > >
> > >     >> >>> > > > > > i don't know the right way to solve this to be
> > > honest, but
> > >     >> >>> > > > > > if you
> > >     >> >>> > > have
> > >     >> >>> > > > > > ideas, i am willing to help.
> > >     >> >>> > > > > >
> > >     >> >>> > > > > >
> > >     >> >>> > > > > >
> > >     >> >>> > > > > > *Will STEVENS*
> > >     >> >>> > > > > > Lead Developer
> > >     >> >>> > > > > >
> > >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
> > >     >> >>> > > > > >
> > >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> > >     >> >>> njohnson@ena.com
> > >     >> >>> > >
> > >     >> >>> > > > > wrote:
> > >     >> >>> > > > > >
> > >     >> >>> > > > > > > So not to re-open a can of worms, but I’m in a
> > > situation
> > >     >> >>> > > > > > > where
> > >     >> >>> I
> > >     >> >>> > > need
> > >     >> >>> > > > > > > to come up with a Marvin component test that
> > > depends on a
> > >     >> >>> > template
> > >     >> >>> > > > > > > based on a kernel that’s relatively new, i.e.,
> > > newer than
> > >     >> >>> Centos
> > >     >> >>> > > 5.3
> > >     >> >>> > > > /
> > >     >> >>> > > > > > Ubuntu 10.04 .
> > >     >> >>> > > > > > > I see openvm.eu has a suitable template (Ubuntu
> > > 16.0.4
> > >     >> >>> > > > > > > for
> > >     >> >>> KVM),
> > >     >> >>> > > but
> > >     >> >>> > > > > > > from looking at the thread "Migrating CloudStack
> > > content
> > >     >> >>> > > > > > > from download.cloud.com” it looks like there is
> > >     >> >>> > > > > > > resistance to using
> > >     >> >>> > > this
> > >     >> >>> > > > at
> > >     >> >>> > > > > > > least for hosting system vm templates over
> > concerns
> > > of
> > >     >> >>> > neutrality.
> > >     >> >>> > > > > > > Would this be suitable for a component test?  If
> > > not,
> > >     >> >>> > > > > > > what is a
> > >     >> >>> > > > > “blessed”
> > >     >> >>> > > > > > > template location?
> > >     >> >>> > > > > > >
> > >     >> >>> > > > > > > Thanks in advance!
> > >     >> >>> > > > > > >
> > >     >> >>> > > > > > > Nathan Johnson
> > >     >> >>> > > > > > > R&D Engineer
> > >     >> >>> > > > > > >
> > >     >> >>> > > > > > >
> > >     >> >>> > > > > > >
> > >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville, TN
> > > 37211
> > >     >> >>> > > > > > > General Office: 615-312-6000
> > >     >> >>> > > > > > >
> > >     >> >>> > > > > > > website | blog | support
> > >     >> >>> > > > > > >
> > >     >> >>> > > > > > >
> > >     >> >>> > > > > > >
> > >     >> >>> > > > > > >
> > >     >> >>> > > > > > >
> > >     >> >>> > > > > > >
> > >     >> >>> > > > > > >
> > >     >> >>> > > > > >
> > >     >> >>> > > > >
> > >     >> >>> > > > >
> > >     >> >>> > > > >
> > >     >> >>> > > > >
> > >     >> >>> > > > >
> > >     >> >>> > > > >
> > >     >> >>> > > > > DISCLAIMER
> > >     >> >>> > > > > ==========
> > >     >> >>> > > > > This e-mail may contain privileged and confidential
> > >     >> >>> > > > > information
> > >     >> >>> which
> > >     >> >>> > > is
> > >     >> >>> > > > > the property of Accelerite, a Persistent Systems
> > > business. It
> > >     >> >>> > > > > is
> > >     >> >>> > > intended
> > >     >> >>> > > > > only for the use of the individual or entity to
> which
> > > it is
> > >     >> >>> > addressed.
> > >     >> >>> > > If
> > >     >> >>> > > > > you are not the intended recipient, you are not
> > > authorized to
> > >     >> >>> > > > > read,
> > >     >> >>> > > > retain,
> > >     >> >>> > > > > copy, print, distribute or use this message. If you
> > have
> > >     >> >>> > > > > received
> > >     >> >>> > this
> > >     >> >>> > > > > communication in error, please notify the sender and
> > > delete
> > >     >> >>> > > > > all
> > >     >> >>> > copies
> > >     >> >>> > > of
> > >     >> >>> > > > > this message. Accelerite, a Persistent Systems
> > business
> > > does
> > >     >> >>> > > > > not
> > >     >> >>> > accept
> > >     >> >>> > > > any
> > >     >> >>> > > > > liability for virus infected mails.
> > >     >> >>> > > > >
> > >     >> >>> > > >
> > >     >> >>> > >
> > >     >> >>> >
> > >     >> >>>
> > >     >> >
> > >     >> >abhinandan.prateek@shapeblue.com
> > >     >> >www.shapeblue.com
> > >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
> > >     >> >
> > >     >> >
> > >     >> >
> > >     >>
> > >     >> abhinandan.prateek@shapeblue.com
> > >     >> www.shapeblue.com
> > >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > >     >> @shapeblue
> > >     >>
> > >     >>
> > >     >>
> > >     >>
> > >     >
> > >
> > >
> > >
> > > daan.hoogland@shapeblue.com
> > > www.shapeblue.com
> > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > @shapeblue
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Rafael Weingärtner
> >
>
>
>
> --
> Rafael Weingärtner
>

Re: Modern template hosting

Posted by Rafael Weingärtner <ra...@gmail.com>.

I think I understood you, but I did not understand you.

Let me see if I can create an example to illustrate how we could do using
the HTTP 300 code.

The HTTP 300 code indicates clients that he/she/it has multiple choices to
access the requested resource. If the request is a HEAD, this is the only
thing that the server returns. If the request if a GET, then the server
returns the choices for the client. The response type is defined on the
server based on the Content-Type, user agent and others.

Let’s say the server receives a GET request and informs the content-type as
JSON (application/JSON). Your application could get the mirrors list, do
the appending process for the requested resource path, and then return a
JSON array with the possible mirror the client can use. Then, the client
gets this list and does the processing required to select a mirror.

Now let’s say that a human access the link using a browser. Then, the
server should receive something like (text/plain) as the content-type; I
think depending on the browser this field may differ a little (it would
require some checking). Anyways, the server detects that it is a “human”
requesting the resource, then we could serve an HTML page with a Javascript
that uses the mirror list. This Javascript could do some checking and
choose the best mirror for that specific.

Why do I think this approach is interesting?
The application you developed would be used only to retrieve valid and
trusted sources of system VM images (a trusted repository for mirrors that
we as a community vouch for).  Also, we shift the decision process
regarding mirrors from the server to the client. Then, it is up to clients
to select mirrors, and not up to us (or some of our implementations).

Did this help?
BTW: I am not saying I am against the way you proposed, which would work
fine. It is merely a suggestion using a different perspective.

On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens <wi...@gmail.com>
wrote:

> @rafael: in general, I don't think the client should ever be making a
> choice about a mirror. We have to assume we are working with a scripted
> application and anything hard coded in that implementation on the client
> side is a risk.
>
> I may not be understanding the use of the 300 approach, so I will research
> it to see if I can make it fit.
>
> On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <ra...@gmail.com>
> wrote:
>
> Will great job.
>
> I had the same doubt as Daan.
>
> Have you considered using HTTP 300 working mode? Then, we could let the
> client decide which mirror is the best(closest?). Section “10.3.1” of [1]
> talks about it; this would put the pressure on deciding upon a mirror on
> the client side, but I think it would be fair.
>
> [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
>
> PS: You really like very short variables names!
>
> On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <daan.hoogland@shapeblue.com
> >
> wrote:
>
> > Nice little thing Will,
> >
> > One question: if I read the code correctly it ‘go’es and tries all
> mirrors
> > at once and whichever responses first is redirected to the client. This
> > might well be the same every time. This might be the one closest to the
> > mirror302 and not to the client. Is that correct or did I miss a weighing
> > algorithm hidden in there?
> >
> > Good coding,
> >
> > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
> >
> >     Hey All,
> >     Please review this repo and tell me what you think:
> >     https://github.com/swill/mirror302
> >
> >     Let me know if you have questions or you would like me to make
> changes
> >     before I bring this topic up with ASF Infra and open a ticket to get
> > this
> >     implemented.
> >
> >     Cheers,
> >
> >     *Will Stevens*
> >
> >
> >
> >     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <wstevens@cloudops.com
> >
> > wrote:
> >
> >     > I am building a short term solution right now.  I hopefully will
> > have the
> >     > building blocks in place by the weekend so we can start working
> with
> > Infra
> >     > to get it in place.
> >     >
> >     > We will have a web server which we can point something like
> >     > downloads.cloudstack.org (or whatever url) at.  Then we will have
> > an ASF
> >     > repo which tracks a mirror list and exposes a static site to be
> able
> > to
> >     > query the mirror list.
> >     >
> >     > The web server will receive a request and will do a 302 redirect to
> > the
> >     > appropriate resource on one of the mirrors.
> >     >
> >     > This gets us started.  Legacy environments will have to do a DB
> > change to
> >     > change from 'download.cloud.com' to 'downloads.cloudstack.org' (or
> >     > whatever), but otherwise it should be pretty simple.
> >     >
> >     > We can improve how we deliver templates going forward, but this
> > obviously
> >     > requires some discussion still.  I will try to buy us some time
> with
> > an
> >     > implementation which solves our problems today...
> >     >
> >     > *Will STEVENS*
> >     > Lead Developer
> >     >
> >     > <https://goo.gl/NYZ8KK>
> >     >
> >     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> >     > abhinandan.prateek@shapeblue.com> wrote:
> >     >
> >     >>
> >     >> Initial seeding is a manual step and that is not going to change
> in
> > near
> >     >> future. A handy list of official places from where these templates
> > can be
> >     >> downloaded won’t harm this part of the installation. Just allow
> > anyone
> >     >> credible who follows democratic process to publish their templates
> > with
> >     >> apache’s blessings.
> >     >>
> >     >>
> >     >> Coming to urls hard coded in db, the popular suggestion is to
> point
> > them
> >     >> to mirrors. This looks good but this is going to take some effort
> > that has
> >     >> not materialised yet. Once this change is made it will be going in
> > some
> >     >> future release like 4.11; the problem still remains for the
> > releases that
> >     >> are already out there. In case access to download.cloud.com is
> > dropped
> >     >> at some point we will have to document the procedure such that
> > people know
> >     >> how to make these stable releases work. That is where the
> procedure
> >     >> documented here https://shankerbalan.net/blog/
> >     >> seed-cloudstack-templates-offline/ should be officially adapted
> > with a
> >     >> marketplace to pick up templates of choosing. Maybe we can have a
> > pre-setup
> >     >> script to make choices instead of written wiki. Note that this
> will
> > be
> >     >> anyway needed by the older releases.
> >     >>
> >     >>
> >     >>
> >     >>
> >     >> On 01/03/17, 4:24 PM, "Paul Angus" <pa...@shapeblue.com>
> > wrote:
> >     >>
> >     >> >-1
> >     >> >
> >     >> >We are trying to get MORE people to use CloudStack.  And raising
> > the
> >     >> barrier to entry does nothing to help that.
> >     >> >
> >     >> >
> >     >> >Kind regards,
> >     >> >
> >     >> >Paul Angus
> >     >> >
> >     >> >paul.angus@shapeblue.com
> >     >> >www.shapeblue.com
> >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >     >> >@shapeblue
> >     >> >
> >     >> >
> >     >> >
> >     >> >
> >     >> >-----Original Message-----
> >     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@
> shapeblue.com]
> >     >> >Sent: 01 March 2017 09:39
> >     >> >To: dev@cloudstack.apache.org
> >     >> >Subject: Re: Modern template hosting
> >     >> >
> >     >> >+1
> >     >> >
> >     >> >After seeing this discussion running in circles several times, I
> > think
> >     >> we should at least get started with the simplest option.
> >     >> >
> >     >> >
> >     >> >
> >     >> >
> >     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <ch...@gmail.com>
> > wrote:
> >     >> >
> >     >> >>The alternative foolproof way is to simply not provide the
> > automatic
> >     >> >>download.
> >     >> >>
> >     >> >>Just document the mirror list URL where the templates can be
> > found. I
> >     >> >>don't know why people reject this option.
> >     >> >>
> >     >> >>https://shankerbalan.net/blog/seed-cloudstack-
> templates-offline/
> >     >> >>
> >     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> >     >> >><wi...@gmail.com>
> >     >> >>wrote:
> >     >> >>
> >     >> >>> OK. Thanks for the heads up.
> >     >> >>>
> >     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > chiradeepv@gmail.com>
> >     >> wrote:
> >     >> >>>
> >     >> >>> > Sounds workable. The downloader code in the SSVM won't
> follow
> >     >> >>> > redirects I think.
> >     >> >>> > https://github.com/apache/cloudstack/blob/
> 5511065fc20787619d
> > 9cd0444
> >     >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
> >     >> >>> > HttpTemplateDownloader.java#L93
> >     >> >>> > https://goo.gl/dSi0r5
> >     >> >>> >  Might need to add
> >     >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
> >     >> >>> >
> >     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> >     >> >>> > <ws...@cloudops.com>
> >     >> >>> > wrote:
> >     >> >>> >
> >     >> >>> > > We haven't opened a ticket yet because we don't have a
> > strategy
> >     >> yet.
> >     >> >>> > >
> >     >> >>> > > What do you guys think of this:
> >     >> >>> > > - We setup a new github repo in the 'apache' org which
> > consists
> >     >> >>> > > of a
> >     >> >>> > single
> >     >> >>> > > file with a list of active/supported mirrors.
> >     >> >>> > > - I write a small web server, distributed as a binary,
> > which can
> >     >> >>> > > be
> >     >> >>> > hosted
> >     >> >>> > > by ASF Infra.  This web server will query the current list
> > of
> >     >> >>> > > mirrors
> >     >> >>> and
> >     >> >>> > > will select one and then do a 302 redirect to that mirror.
> >     >> >>> > >
> >     >> >>> > > The act of 'choosing' a mirror could be done in a number
> of
> > ways.
> >     >> >>> > > - If we want to define an order, then it could just try
> > from the
> >     >> >>> > > top of
> >     >> >>> > the
> >     >> >>> > > list and work its way down.  It would curl the target to
> > make
> >     >> >>> > > sure it
> >     >> >>> > gets
> >     >> >>> > > a 200 and if it does, it would do a 302 redirect.
> >     >> >>> > > - Or, if we want to distribute the load across the
> mirrors,
> > we
> >     >> >>> > > could
> >     >> >>> pick
> >     >> >>> > > from the list randomly.  Again, doing a curl to verify the
> > mirror
> >     >> >>> > > is up
> >     >> >>> > and
> >     >> >>> > > then doing a redirect.
> >     >> >>> > > - If we want to get fancy, we could do a reverse IP lookup
> > and
> >     >> >>> > > try to
> >     >> >>> > match
> >     >> >>> > > the requester with their closest geographical mirror.
> >     >> >>> > >
> >     >> >>> > > Thoughts?
> >     >> >>> > >
> >     >> >>> > > *Will STEVENS*
> >     >> >>> > > Lead Developer
> >     >> >>> > >
> >     >> >>> > > <https://goo.gl/NYZ8KK>
> >     >> >>> > >
> >     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> >     >> >>> chiradeepv@gmail.com
> >     >> >>> > >
> >     >> >>> > > wrote:
> >     >> >>> > >
> >     >> >>> > > > What steps are needed to set up a mirror? What does
> Infra
> > need
> >     >> to do?
> >     >> >>> > Has
> >     >> >>> > > > anybody filed a ticket with Infra?
> >     >> >>> > > >
> >     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> >     >> >>> > > > raja.pullela@accelerite.com>
> >     >> >>> > > > wrote:
> >     >> >>> > > >
> >     >> >>> > > > > Hi will,
> >     >> >>> > > > >
> >     >> >>> > > > > I believe, we didn’t get to close ‘getting a mirror on
> > Apache’
> >     >> >>> > because
> >     >> >>> > > we
> >     >> >>> > > > > needed someone on the Apache Infra side to close this.
> > BTW,
> >     >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
> >     >> >>> > > > > all/most of
> >     >> >>> > the
> >     >> >>> > > > > content.  Once we can close on the Apache mirror for
> > hosting
> >     >> >>> > > > > the
> >     >> >>> > > > content, I
> >     >> >>> > > > > can help assist getting the content there.
> >     >> >>> > > > >
> >     >> >>> > > > > For now, we have replicated the download.cloud.com
> > content to
> >     >> >>> > > > > ‘ s3.download.accelerite.com’.
> >     >> >>> > > > > Also, we are working on a set of steps/procedure to
> > help with
> >     >> >>> > > > > this change.  I will update everyone in about a week’s
> > time
> >     >> >>> > > > > on the
> >     >> >>> > details.
> >     >> >>> > > > >
> >     >> >>> > > > > Best,
> >     >> >>> > > > > Raja Pullela
> >     >> >>> > > > > Engineering Team,
> >     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA,
> 95054
> >     >> >>> > > > >
> >     >> >>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on
> > behalf of
> >     >> >>> > > > > Will Stevens" <williamstevens@gmail.com on behalf of
> >     >> >>> > wstevens@cloudops.com>
> >     >> >>> > > > > wrote:
> >     >> >>> > > > >
> >     >> >>> > > > > unfortunately the template mirror conversation got
> > caught up
> >     >> >>> > > > > in
> >     >> >>> > details
> >     >> >>> > > > and
> >     >> >>> > > > > nobody took the lead on implementing a solution.
> >     >> >>> > > > >
> >     >> >>> > > > > citrix has been pinging me every couple months to say
> > 'dude,
> >     >> >>> > > > > we
> >     >> >>> need
> >     >> >>> > to
> >     >> >>> > > > > remove the dependency on download.citrix.com', but i
> > have not
> >     >> >>> > > > > had
> >     >> >>> > the
> >     >> >>> > > > > cycles to get in and solve the problem.  the shutdown
> > of that
> >     >> >>> > > > > is
> >     >> >>> > > imminent
> >     >> >>> > > > > right now, so we need to solve it asap.
> >     >> >>> > > > >
> >     >> >>> > > > >
> >     >> >>> > > > >
> >     >> >>> > > > > *Will STEVENS*
> >     >> >>> > > > > Lead Developer
> >     >> >>> > > > >
> >     >> >>> > > > > <https://goo.gl/NYZ8KK>
> >     >> >>> > > > >
> >     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> >     >> >>> > paul.angus@shapeblue.com
> >     >> >>> > > >
> >     >> >>> > > > > wrote:
> >     >> >>> > > > >
> >     >> >>> > > > > > Hi Nathan,
> >     >> >>> > > > > >
> >     >> >>> > > > > > Ideally, if you put the template location in (or use
> a
> >     >> >>> > > > > > template
> >     >> >>> > > defined
> >     >> >>> > > > > > in)  test_data.py then the actual location can be
> >     >> >>> > > > > > overridden by
> >     >> >>> > > anyone
> >     >> >>> > > > > > testing.
> >     >> >>> > > > > >
> >     >> >>> > > > > > For Trillian, we've copied all of the templates that
> > people
> >     >> >>> > > > > > have
> >     >> >>> > > define
> >     >> >>> > > > > to
> >     >> >>> > > > > > a local repo and then replace the URLs in
> > test_data.py to
> >     >> >>> > > > > > reduce
> >     >> >>> > > > > bandwidth
> >     >> >>> > > > > > use and download times.
> >     >> >>> > > > > >
> >     >> >>> > > > > > Ie:
> >     >> >>> > > > > >
> >     >> >>> > > > > >             "bootableIso":
> >     >> >>> > > > > >                 {
> >     >> >>> > > > > >                     "displaytext": "Test Bootable
> > ISO",
> >     >> >>> > > > > >                     "name": "testISO",
> >     >> >>> > > > > >                     "bootable": True,
> >     >> >>> > > > > >                     "ispublic": False,
> >     >> >>> > > > > >                     "url": "{{
> marvin_images_location
> >     >> >>> > > > > > }}/TinyCore-current.iso",
> >     >> >>> > > > > >                     "ostype": 'Other Linux
> (64-bit)',
> >     >> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
> >     >> >>> > > > > >         },
> >     >> >>> > > > > >
> >     >> >>> > > > > >
> >     >> >>> > > > > >
> >     >> >>> > > > > > I thought that we had come up with a solution for
> >     >> >>> > download.cloud.com
> >     >> >>> > > ,
> >     >> >>> > > > by
> >     >> >>> > > > > > having a mirrorlist hosted in Community Apache
> > 'space' with
> >     >> >>> anyone
> >     >> >>> > > able
> >     >> >>> > > > > to
> >     >> >>> > > > > > out themselves forward as a mirror.
> >     >> >>> > > > > > But I must admit I lost track of whether anyone made
> > the
> >     >> >>> requisite
> >     >> >>> > > > > changes
> >     >> >>> > > > > > in code....
> >     >> >>> > > > > >
> >     >> >>> > > > > >
> >     >> >>> > > > > >
> >     >> >>> > > > > >
> >     >> >>> > > > > > Kind regards,
> >     >> >>> > > > > >
> >     >> >>> > > > > > Paul Angus
> >     >> >>> > > > > >
> >     >> >>> > > > > > paul.angus@shapeblue.com
> >     >> >>> > > > > > www.shapeblue.com
> >     >> >>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >     >> >>> > > > > > @shapeblue
> >     >> >>> > > > > >
> >     >> >>> > > > > >
> >     >> >>> > > > > >
> >     >> >>> > > > > >
> >     >> >>> > > > > > -----Original Message-----
> >     >> >>> > > > > > From: williamstevens@gmail.com
> >     >> >>> > > > > > [mailto:williamstevens@gmail.com]
> >     >> >>> > On
> >     >> >>> > > > > > Behalf Of Will Stevens
> >     >> >>> > > > > > Sent: 24 February 2017 16:30
> >     >> >>> > > > > > To: dev@cloudstack.apache.org
> >     >> >>> > > > > > Subject: Re: Modern template hosting
> >     >> >>> > > > > >
> >     >> >>> > > > > > this is a hard questions.  in general, we should be
> > setting
> >     >> >>> > > > > > up a
> >     >> >>> > > mirror
> >     >> >>> > > > > on
> >     >> >>> > > > > > some cloudstack/apache domain and then mirror to
> other
> >     >> >>> > > > > > provided
> >     >> >>> > > > > templates.
> >     >> >>> > > > > >
> >     >> >>> > > > > > we MUST come up with a solution to deprecate
> >     >> >>> > > > > > 'download.cloud.com
> >     >> >>> ',
> >     >> >>> > > > that
> >     >> >>> > > > > > is going to be going away any day now.
> >     >> >>> > > > > >
> >     >> >>> > > > > > i don't know the right way to solve this to be
> > honest, but
> >     >> >>> > > > > > if you
> >     >> >>> > > have
> >     >> >>> > > > > > ideas, i am willing to help.
> >     >> >>> > > > > >
> >     >> >>> > > > > >
> >     >> >>> > > > > >
> >     >> >>> > > > > > *Will STEVENS*
> >     >> >>> > > > > > Lead Developer
> >     >> >>> > > > > >
> >     >> >>> > > > > > <https://goo.gl/NYZ8KK>
> >     >> >>> > > > > >
> >     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> >     >> >>> njohnson@ena.com
> >     >> >>> > >
> >     >> >>> > > > > wrote:
> >     >> >>> > > > > >
> >     >> >>> > > > > > > So not to re-open a can of worms, but I’m in a
> > situation
> >     >> >>> > > > > > > where
> >     >> >>> I
> >     >> >>> > > need
> >     >> >>> > > > > > > to come up with a Marvin component test that
> > depends on a
> >     >> >>> > template
> >     >> >>> > > > > > > based on a kernel that’s relatively new, i.e.,
> > newer than
> >     >> >>> Centos
> >     >> >>> > > 5.3
> >     >> >>> > > > /
> >     >> >>> > > > > > Ubuntu 10.04 .
> >     >> >>> > > > > > > I see openvm.eu has a suitable template (Ubuntu
> > 16.0.4
> >     >> >>> > > > > > > for
> >     >> >>> KVM),
> >     >> >>> > > but
> >     >> >>> > > > > > > from looking at the thread "Migrating CloudStack
> > content
> >     >> >>> > > > > > > from download.cloud.com” it looks like there is
> >     >> >>> > > > > > > resistance to using
> >     >> >>> > > this
> >     >> >>> > > > at
> >     >> >>> > > > > > > least for hosting system vm templates over
> concerns
> > of
> >     >> >>> > neutrality.
> >     >> >>> > > > > > > Would this be suitable for a component test?  If
> > not,
> >     >> >>> > > > > > > what is a
> >     >> >>> > > > > “blessed”
> >     >> >>> > > > > > > template location?
> >     >> >>> > > > > > >
> >     >> >>> > > > > > > Thanks in advance!
> >     >> >>> > > > > > >
> >     >> >>> > > > > > > Nathan Johnson
> >     >> >>> > > > > > > R&D Engineer
> >     >> >>> > > > > > >
> >     >> >>> > > > > > >
> >     >> >>> > > > > > >
> >     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville, TN
> > 37211
> >     >> >>> > > > > > > General Office: 615-312-6000
> >     >> >>> > > > > > >
> >     >> >>> > > > > > > website | blog | support
> >     >> >>> > > > > > >
> >     >> >>> > > > > > >
> >     >> >>> > > > > > >
> >     >> >>> > > > > > >
> >     >> >>> > > > > > >
> >     >> >>> > > > > > >
> >     >> >>> > > > > > >
> >     >> >>> > > > > >
> >     >> >>> > > > >
> >     >> >>> > > > >
> >     >> >>> > > > >
> >     >> >>> > > > >
> >     >> >>> > > > >
> >     >> >>> > > > >
> >     >> >>> > > > > DISCLAIMER
> >     >> >>> > > > > ==========
> >     >> >>> > > > > This e-mail may contain privileged and confidential
> >     >> >>> > > > > information
> >     >> >>> which
> >     >> >>> > > is
> >     >> >>> > > > > the property of Accelerite, a Persistent Systems
> > business. It
> >     >> >>> > > > > is
> >     >> >>> > > intended
> >     >> >>> > > > > only for the use of the individual or entity to which
> > it is
> >     >> >>> > addressed.
> >     >> >>> > > If
> >     >> >>> > > > > you are not the intended recipient, you are not
> > authorized to
> >     >> >>> > > > > read,
> >     >> >>> > > > retain,
> >     >> >>> > > > > copy, print, distribute or use this message. If you
> have
> >     >> >>> > > > > received
> >     >> >>> > this
> >     >> >>> > > > > communication in error, please notify the sender and
> > delete
> >     >> >>> > > > > all
> >     >> >>> > copies
> >     >> >>> > > of
> >     >> >>> > > > > this message. Accelerite, a Persistent Systems
> business
> > does
> >     >> >>> > > > > not
> >     >> >>> > accept
> >     >> >>> > > > any
> >     >> >>> > > > > liability for virus infected mails.
> >     >> >>> > > > >
> >     >> >>> > > >
> >     >> >>> > >
> >     >> >>> >
> >     >> >>>
> >     >> >
> >     >> >abhinandan.prateek@shapeblue.com
> >     >> >www.shapeblue.com
> >     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
> >     >> >
> >     >> >
> >     >> >
> >     >>
> >     >> abhinandan.prateek@shapeblue.com
> >     >> www.shapeblue.com
> >     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >     >> @shapeblue
> >     >>
> >     >>
> >     >>
> >     >>
> >     >
> >
> >
> >
> > daan.hoogland@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
>
>
> --
> Rafael Weingärtner
>



-- 
Rafael Weingärtner

Re: Modern template hosting

Posted by Will Stevens <wi...@gmail.com>.

@rafael: in general, I don't think the client should ever be making a
choice about a mirror. We have to assume we are working with a scripted
application and anything hard coded in that implementation on the client
side is a risk.

I may not be understanding the use of the 300 approach, so I will research
it to see if I can make it fit.

On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <ra...@gmail.com>
wrote:

Will great job.

I had the same doubt as Daan.

Have you considered using HTTP 300 working mode? Then, we could let the
client decide which mirror is the best(closest?). Section “10.3.1” of [1]
talks about it; this would put the pressure on deciding upon a mirror on
the client side, but I think it would be fair.

[1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

PS: You really like very short variables names!

On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <da...@shapeblue.com>
wrote:

> Nice little thing Will,
>
> One question: if I read the code correctly it ‘go’es and tries all mirrors
> at once and whichever responses first is redirected to the client. This
> might well be the same every time. This might be the one closest to the
> mirror302 and not to the client. Is that correct or did I miss a weighing
> algorithm hidden in there?
>
> Good coding,
>
> On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
>
>     Hey All,
>     Please review this repo and tell me what you think:
>     https://github.com/swill/mirror302
>
>     Let me know if you have questions or you would like me to make changes
>     before I bring this topic up with ASF Infra and open a ticket to get
> this
>     implemented.
>
>     Cheers,
>
>     *Will Stevens*
>
>
>
>     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <ws...@cloudops.com>
> wrote:
>
>     > I am building a short term solution right now.  I hopefully will
> have the
>     > building blocks in place by the weekend so we can start working with
> Infra
>     > to get it in place.
>     >
>     > We will have a web server which we can point something like
>     > downloads.cloudstack.org (or whatever url) at.  Then we will have
> an ASF
>     > repo which tracks a mirror list and exposes a static site to be able
> to
>     > query the mirror list.
>     >
>     > The web server will receive a request and will do a 302 redirect to
> the
>     > appropriate resource on one of the mirrors.
>     >
>     > This gets us started.  Legacy environments will have to do a DB
> change to
>     > change from 'download.cloud.com' to 'downloads.cloudstack.org' (or
>     > whatever), but otherwise it should be pretty simple.
>     >
>     > We can improve how we deliver templates going forward, but this
> obviously
>     > requires some discussion still.  I will try to buy us some time with
> an
>     > implementation which solves our problems today...
>     >
>     > *Will STEVENS*
>     > Lead Developer
>     >
>     > <https://goo.gl/NYZ8KK>
>     >
>     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
>     > abhinandan.prateek@shapeblue.com> wrote:
>     >
>     >>
>     >> Initial seeding is a manual step and that is not going to change in
> near
>     >> future. A handy list of official places from where these templates
> can be
>     >> downloaded won’t harm this part of the installation. Just allow
> anyone
>     >> credible who follows democratic process to publish their templates
> with
>     >> apache’s blessings.
>     >>
>     >>
>     >> Coming to urls hard coded in db, the popular suggestion is to point
> them
>     >> to mirrors. This looks good but this is going to take some effort
> that has
>     >> not materialised yet. Once this change is made it will be going in
> some
>     >> future release like 4.11; the problem still remains for the
> releases that
>     >> are already out there. In case access to download.cloud.com is
> dropped
>     >> at some point we will have to document the procedure such that
> people know
>     >> how to make these stable releases work. That is where the procedure
>     >> documented here https://shankerbalan.net/blog/
>     >> seed-cloudstack-templates-offline/ should be officially adapted
> with a
>     >> marketplace to pick up templates of choosing. Maybe we can have a
> pre-setup
>     >> script to make choices instead of written wiki. Note that this will
> be
>     >> anyway needed by the older releases.
>     >>
>     >>
>     >>
>     >>
>     >> On 01/03/17, 4:24 PM, "Paul Angus" <pa...@shapeblue.com>
> wrote:
>     >>
>     >> >-1
>     >> >
>     >> >We are trying to get MORE people to use CloudStack.  And raising
> the
>     >> barrier to entry does nothing to help that.
>     >> >
>     >> >
>     >> >Kind regards,
>     >> >
>     >> >Paul Angus
>     >> >
>     >> >paul.angus@shapeblue.com
>     >> >www.shapeblue.com
>     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     >> >@shapeblue
>     >> >
>     >> >
>     >> >
>     >> >
>     >> >-----Original Message-----
>     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@shapeblue.com]
>     >> >Sent: 01 March 2017 09:39
>     >> >To: dev@cloudstack.apache.org
>     >> >Subject: Re: Modern template hosting
>     >> >
>     >> >+1
>     >> >
>     >> >After seeing this discussion running in circles several times, I
> think
>     >> we should at least get started with the simplest option.
>     >> >
>     >> >
>     >> >
>     >> >
>     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <ch...@gmail.com>
> wrote:
>     >> >
>     >> >>The alternative foolproof way is to simply not provide the
> automatic
>     >> >>download.
>     >> >>
>     >> >>Just document the mirror list URL where the templates can be
> found. I
>     >> >>don't know why people reject this option.
>     >> >>
>     >> >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
>     >> >>
>     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
>     >> >><wi...@gmail.com>
>     >> >>wrote:
>     >> >>
>     >> >>> OK. Thanks for the heads up.
>     >> >>>
>     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> chiradeepv@gmail.com>
>     >> wrote:
>     >> >>>
>     >> >>> > Sounds workable. The downloader code in the SSVM won't follow
>     >> >>> > redirects I think.
>     >> >>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d
> 9cd0444
>     >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
>     >> >>> > HttpTemplateDownloader.java#L93
>     >> >>> > https://goo.gl/dSi0r5
>     >> >>> >  Might need to add
>     >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
>     >> >>> >
>     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
>     >> >>> > <ws...@cloudops.com>
>     >> >>> > wrote:
>     >> >>> >
>     >> >>> > > We haven't opened a ticket yet because we don't have a
> strategy
>     >> yet.
>     >> >>> > >
>     >> >>> > > What do you guys think of this:
>     >> >>> > > - We setup a new github repo in the 'apache' org which
> consists
>     >> >>> > > of a
>     >> >>> > single
>     >> >>> > > file with a list of active/supported mirrors.
>     >> >>> > > - I write a small web server, distributed as a binary,
> which can
>     >> >>> > > be
>     >> >>> > hosted
>     >> >>> > > by ASF Infra.  This web server will query the current list
> of
>     >> >>> > > mirrors
>     >> >>> and
>     >> >>> > > will select one and then do a 302 redirect to that mirror.
>     >> >>> > >
>     >> >>> > > The act of 'choosing' a mirror could be done in a number of
> ways.
>     >> >>> > > - If we want to define an order, then it could just try
> from the
>     >> >>> > > top of
>     >> >>> > the
>     >> >>> > > list and work its way down.  It would curl the target to
> make
>     >> >>> > > sure it
>     >> >>> > gets
>     >> >>> > > a 200 and if it does, it would do a 302 redirect.
>     >> >>> > > - Or, if we want to distribute the load across the mirrors,
> we
>     >> >>> > > could
>     >> >>> pick
>     >> >>> > > from the list randomly.  Again, doing a curl to verify the
> mirror
>     >> >>> > > is up
>     >> >>> > and
>     >> >>> > > then doing a redirect.
>     >> >>> > > - If we want to get fancy, we could do a reverse IP lookup
> and
>     >> >>> > > try to
>     >> >>> > match
>     >> >>> > > the requester with their closest geographical mirror.
>     >> >>> > >
>     >> >>> > > Thoughts?
>     >> >>> > >
>     >> >>> > > *Will STEVENS*
>     >> >>> > > Lead Developer
>     >> >>> > >
>     >> >>> > > <https://goo.gl/NYZ8KK>
>     >> >>> > >
>     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
>     >> >>> chiradeepv@gmail.com
>     >> >>> > >
>     >> >>> > > wrote:
>     >> >>> > >
>     >> >>> > > > What steps are needed to set up a mirror? What does Infra
> need
>     >> to do?
>     >> >>> > Has
>     >> >>> > > > anybody filed a ticket with Infra?
>     >> >>> > > >
>     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
>     >> >>> > > > raja.pullela@accelerite.com>
>     >> >>> > > > wrote:
>     >> >>> > > >
>     >> >>> > > > > Hi will,
>     >> >>> > > > >
>     >> >>> > > > > I believe, we didn’t get to close ‘getting a mirror on
> Apache’
>     >> >>> > because
>     >> >>> > > we
>     >> >>> > > > > needed someone on the Apache Infra side to close this.
> BTW,
>     >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
>     >> >>> > > > > all/most of
>     >> >>> > the
>     >> >>> > > > > content.  Once we can close on the Apache mirror for
> hosting
>     >> >>> > > > > the
>     >> >>> > > > content, I
>     >> >>> > > > > can help assist getting the content there.
>     >> >>> > > > >
>     >> >>> > > > > For now, we have replicated the download.cloud.com
> content to
>     >> >>> > > > > ‘ s3.download.accelerite.com’.
>     >> >>> > > > > Also, we are working on a set of steps/procedure to
> help with
>     >> >>> > > > > this change.  I will update everyone in about a week’s
> time
>     >> >>> > > > > on the
>     >> >>> > details.
>     >> >>> > > > >
>     >> >>> > > > > Best,
>     >> >>> > > > > Raja Pullela
>     >> >>> > > > > Engineering Team,
>     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA,
95054
>     >> >>> > > > >
>     >> >>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on
> behalf of
>     >> >>> > > > > Will Stevens" <williamstevens@gmail.com on behalf of
>     >> >>> > wstevens@cloudops.com>
>     >> >>> > > > > wrote:
>     >> >>> > > > >
>     >> >>> > > > > unfortunately the template mirror conversation got
> caught up
>     >> >>> > > > > in
>     >> >>> > details
>     >> >>> > > > and
>     >> >>> > > > > nobody took the lead on implementing a solution.
>     >> >>> > > > >
>     >> >>> > > > > citrix has been pinging me every couple months to say
> 'dude,
>     >> >>> > > > > we
>     >> >>> need
>     >> >>> > to
>     >> >>> > > > > remove the dependency on download.citrix.com', but i
> have not
>     >> >>> > > > > had
>     >> >>> > the
>     >> >>> > > > > cycles to get in and solve the problem.  the shutdown
> of that
>     >> >>> > > > > is
>     >> >>> > > imminent
>     >> >>> > > > > right now, so we need to solve it asap.
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > > *Will STEVENS*
>     >> >>> > > > > Lead Developer
>     >> >>> > > > >
>     >> >>> > > > > <https://goo.gl/NYZ8KK>
>     >> >>> > > > >
>     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
>     >> >>> > paul.angus@shapeblue.com
>     >> >>> > > >
>     >> >>> > > > > wrote:
>     >> >>> > > > >
>     >> >>> > > > > > Hi Nathan,
>     >> >>> > > > > >
>     >> >>> > > > > > Ideally, if you put the template location in (or use
a
>     >> >>> > > > > > template
>     >> >>> > > defined
>     >> >>> > > > > > in)  test_data.py then the actual location can be
>     >> >>> > > > > > overridden by
>     >> >>> > > anyone
>     >> >>> > > > > > testing.
>     >> >>> > > > > >
>     >> >>> > > > > > For Trillian, we've copied all of the templates that
> people
>     >> >>> > > > > > have
>     >> >>> > > define
>     >> >>> > > > > to
>     >> >>> > > > > > a local repo and then replace the URLs in
> test_data.py to
>     >> >>> > > > > > reduce
>     >> >>> > > > > bandwidth
>     >> >>> > > > > > use and download times.
>     >> >>> > > > > >
>     >> >>> > > > > > Ie:
>     >> >>> > > > > >
>     >> >>> > > > > >             "bootableIso":
>     >> >>> > > > > >                 {
>     >> >>> > > > > >                     "displaytext": "Test Bootable
> ISO",
>     >> >>> > > > > >                     "name": "testISO",
>     >> >>> > > > > >                     "bootable": True,
>     >> >>> > > > > >                     "ispublic": False,
>     >> >>> > > > > >                     "url": "{{ marvin_images_location
>     >> >>> > > > > > }}/TinyCore-current.iso",
>     >> >>> > > > > >                     "ostype": 'Other Linux (64-bit)',
>     >> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
>     >> >>> > > > > >         },
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > I thought that we had come up with a solution for
>     >> >>> > download.cloud.com
>     >> >>> > > ,
>     >> >>> > > > by
>     >> >>> > > > > > having a mirrorlist hosted in Community Apache
> 'space' with
>     >> >>> anyone
>     >> >>> > > able
>     >> >>> > > > > to
>     >> >>> > > > > > out themselves forward as a mirror.
>     >> >>> > > > > > But I must admit I lost track of whether anyone made
> the
>     >> >>> requisite
>     >> >>> > > > > changes
>     >> >>> > > > > > in code....
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > Kind regards,
>     >> >>> > > > > >
>     >> >>> > > > > > Paul Angus
>     >> >>> > > > > >
>     >> >>> > > > > > paul.angus@shapeblue.com
>     >> >>> > > > > > www.shapeblue.com
>     >> >>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     >> >>> > > > > > @shapeblue
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > -----Original Message-----
>     >> >>> > > > > > From: williamstevens@gmail.com
>     >> >>> > > > > > [mailto:williamstevens@gmail.com]
>     >> >>> > On
>     >> >>> > > > > > Behalf Of Will Stevens
>     >> >>> > > > > > Sent: 24 February 2017 16:30
>     >> >>> > > > > > To: dev@cloudstack.apache.org
>     >> >>> > > > > > Subject: Re: Modern template hosting
>     >> >>> > > > > >
>     >> >>> > > > > > this is a hard questions.  in general, we should be
> setting
>     >> >>> > > > > > up a
>     >> >>> > > mirror
>     >> >>> > > > > on
>     >> >>> > > > > > some cloudstack/apache domain and then mirror to
other
>     >> >>> > > > > > provided
>     >> >>> > > > > templates.
>     >> >>> > > > > >
>     >> >>> > > > > > we MUST come up with a solution to deprecate
>     >> >>> > > > > > 'download.cloud.com
>     >> >>> ',
>     >> >>> > > > that
>     >> >>> > > > > > is going to be going away any day now.
>     >> >>> > > > > >
>     >> >>> > > > > > i don't know the right way to solve this to be
> honest, but
>     >> >>> > > > > > if you
>     >> >>> > > have
>     >> >>> > > > > > ideas, i am willing to help.
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > *Will STEVENS*
>     >> >>> > > > > > Lead Developer
>     >> >>> > > > > >
>     >> >>> > > > > > <https://goo.gl/NYZ8KK>
>     >> >>> > > > > >
>     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
>     >> >>> njohnson@ena.com
>     >> >>> > >
>     >> >>> > > > > wrote:
>     >> >>> > > > > >
>     >> >>> > > > > > > So not to re-open a can of worms, but I’m in a
> situation
>     >> >>> > > > > > > where
>     >> >>> I
>     >> >>> > > need
>     >> >>> > > > > > > to come up with a Marvin component test that
> depends on a
>     >> >>> > template
>     >> >>> > > > > > > based on a kernel that’s relatively new, i.e.,
> newer than
>     >> >>> Centos
>     >> >>> > > 5.3
>     >> >>> > > > /
>     >> >>> > > > > > Ubuntu 10.04 .
>     >> >>> > > > > > > I see openvm.eu has a suitable template (Ubuntu
> 16.0.4
>     >> >>> > > > > > > for
>     >> >>> KVM),
>     >> >>> > > but
>     >> >>> > > > > > > from looking at the thread "Migrating CloudStack
> content
>     >> >>> > > > > > > from download.cloud.com” it looks like there is
>     >> >>> > > > > > > resistance to using
>     >> >>> > > this
>     >> >>> > > > at
>     >> >>> > > > > > > least for hosting system vm templates over concerns
> of
>     >> >>> > neutrality.
>     >> >>> > > > > > > Would this be suitable for a component test?  If
> not,
>     >> >>> > > > > > > what is a
>     >> >>> > > > > “blessed”
>     >> >>> > > > > > > template location?
>     >> >>> > > > > > >
>     >> >>> > > > > > > Thanks in advance!
>     >> >>> > > > > > >
>     >> >>> > > > > > > Nathan Johnson
>     >> >>> > > > > > > R&D Engineer
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville, TN
> 37211
>     >> >>> > > > > > > General Office: 615-312-6000
>     >> >>> > > > > > >
>     >> >>> > > > > > > website | blog | support
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > > DISCLAIMER
>     >> >>> > > > > ==========
>     >> >>> > > > > This e-mail may contain privileged and confidential
>     >> >>> > > > > information
>     >> >>> which
>     >> >>> > > is
>     >> >>> > > > > the property of Accelerite, a Persistent Systems
> business. It
>     >> >>> > > > > is
>     >> >>> > > intended
>     >> >>> > > > > only for the use of the individual or entity to which
> it is
>     >> >>> > addressed.
>     >> >>> > > If
>     >> >>> > > > > you are not the intended recipient, you are not
> authorized to
>     >> >>> > > > > read,
>     >> >>> > > > retain,
>     >> >>> > > > > copy, print, distribute or use this message. If you
have
>     >> >>> > > > > received
>     >> >>> > this
>     >> >>> > > > > communication in error, please notify the sender and
> delete
>     >> >>> > > > > all
>     >> >>> > copies
>     >> >>> > > of
>     >> >>> > > > > this message. Accelerite, a Persistent Systems business
> does
>     >> >>> > > > > not
>     >> >>> > accept
>     >> >>> > > > any
>     >> >>> > > > > liability for virus infected mails.
>     >> >>> > > > >
>     >> >>> > > >
>     >> >>> > >
>     >> >>> >
>     >> >>>
>     >> >
>     >> >abhinandan.prateek@shapeblue.com
>     >> >www.shapeblue.com
>     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>     >> >
>     >> >
>     >> >
>     >>
>     >> abhinandan.prateek@shapeblue.com
>     >> www.shapeblue.com
>     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     >> @shapeblue
>     >>
>     >>
>     >>
>     >>
>     >
>
>
>
> daan.hoogland@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>


--
Rafael Weingärtner

Re: Modern template hosting

Posted by Will Stevens <wi...@gmail.com>.

I can look at the 300 approach. Thanks for the suggestion.

Haha. Ya, I meant to go back and expand the variables in the inline
function. Old habit when using inline functions so it is easier to see them
when scanning the code. I need to stop doing that. :)

On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <ra...@gmail.com>
wrote:

Will great job.

I had the same doubt as Daan.

Have you considered using HTTP 300 working mode? Then, we could let the
client decide which mirror is the best(closest?). Section “10.3.1” of [1]
talks about it; this would put the pressure on deciding upon a mirror on
the client side, but I think it would be fair.

[1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

PS: You really like very short variables names!

On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <da...@shapeblue.com>
wrote:

> Nice little thing Will,
>
> One question: if I read the code correctly it ‘go’es and tries all mirrors
> at once and whichever responses first is redirected to the client. This
> might well be the same every time. This might be the one closest to the
> mirror302 and not to the client. Is that correct or did I miss a weighing
> algorithm hidden in there?
>
> Good coding,
>
> On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
>
>     Hey All,
>     Please review this repo and tell me what you think:
>     https://github.com/swill/mirror302
>
>     Let me know if you have questions or you would like me to make changes
>     before I bring this topic up with ASF Infra and open a ticket to get
> this
>     implemented.
>
>     Cheers,
>
>     *Will Stevens*
>
>
>
>     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <ws...@cloudops.com>
> wrote:
>
>     > I am building a short term solution right now.  I hopefully will
> have the
>     > building blocks in place by the weekend so we can start working with
> Infra
>     > to get it in place.
>     >
>     > We will have a web server which we can point something like
>     > downloads.cloudstack.org (or whatever url) at.  Then we will have
> an ASF
>     > repo which tracks a mirror list and exposes a static site to be able
> to
>     > query the mirror list.
>     >
>     > The web server will receive a request and will do a 302 redirect to
> the
>     > appropriate resource on one of the mirrors.
>     >
>     > This gets us started.  Legacy environments will have to do a DB
> change to
>     > change from 'download.cloud.com' to 'downloads.cloudstack.org' (or
>     > whatever), but otherwise it should be pretty simple.
>     >
>     > We can improve how we deliver templates going forward, but this
> obviously
>     > requires some discussion still.  I will try to buy us some time with
> an
>     > implementation which solves our problems today...
>     >
>     > *Will STEVENS*
>     > Lead Developer
>     >
>     > <https://goo.gl/NYZ8KK>
>     >
>     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
>     > abhinandan.prateek@shapeblue.com> wrote:
>     >
>     >>
>     >> Initial seeding is a manual step and that is not going to change in
> near
>     >> future. A handy list of official places from where these templates
> can be
>     >> downloaded won’t harm this part of the installation. Just allow
> anyone
>     >> credible who follows democratic process to publish their templates
> with
>     >> apache’s blessings.
>     >>
>     >>
>     >> Coming to urls hard coded in db, the popular suggestion is to point
> them
>     >> to mirrors. This looks good but this is going to take some effort
> that has
>     >> not materialised yet. Once this change is made it will be going in
> some
>     >> future release like 4.11; the problem still remains for the
> releases that
>     >> are already out there. In case access to download.cloud.com is
> dropped
>     >> at some point we will have to document the procedure such that
> people know
>     >> how to make these stable releases work. That is where the procedure
>     >> documented here https://shankerbalan.net/blog/
>     >> seed-cloudstack-templates-offline/ should be officially adapted
> with a
>     >> marketplace to pick up templates of choosing. Maybe we can have a
> pre-setup
>     >> script to make choices instead of written wiki. Note that this will
> be
>     >> anyway needed by the older releases.
>     >>
>     >>
>     >>
>     >>
>     >> On 01/03/17, 4:24 PM, "Paul Angus" <pa...@shapeblue.com>
> wrote:
>     >>
>     >> >-1
>     >> >
>     >> >We are trying to get MORE people to use CloudStack.  And raising
> the
>     >> barrier to entry does nothing to help that.
>     >> >
>     >> >
>     >> >Kind regards,
>     >> >
>     >> >Paul Angus
>     >> >
>     >> >paul.angus@shapeblue.com
>     >> >www.shapeblue.com
>     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     >> >@shapeblue
>     >> >
>     >> >
>     >> >
>     >> >
>     >> >-----Original Message-----
>     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@shapeblue.com]
>     >> >Sent: 01 March 2017 09:39
>     >> >To: dev@cloudstack.apache.org
>     >> >Subject: Re: Modern template hosting
>     >> >
>     >> >+1
>     >> >
>     >> >After seeing this discussion running in circles several times, I
> think
>     >> we should at least get started with the simplest option.
>     >> >
>     >> >
>     >> >
>     >> >
>     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <ch...@gmail.com>
> wrote:
>     >> >
>     >> >>The alternative foolproof way is to simply not provide the
> automatic
>     >> >>download.
>     >> >>
>     >> >>Just document the mirror list URL where the templates can be
> found. I
>     >> >>don't know why people reject this option.
>     >> >>
>     >> >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
>     >> >>
>     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
>     >> >><wi...@gmail.com>
>     >> >>wrote:
>     >> >>
>     >> >>> OK. Thanks for the heads up.
>     >> >>>
>     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> chiradeepv@gmail.com>
>     >> wrote:
>     >> >>>
>     >> >>> > Sounds workable. The downloader code in the SSVM won't follow
>     >> >>> > redirects I think.
>     >> >>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d
> 9cd0444
>     >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
>     >> >>> > HttpTemplateDownloader.java#L93
>     >> >>> > https://goo.gl/dSi0r5
>     >> >>> >  Might need to add
>     >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
>     >> >>> >
>     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
>     >> >>> > <ws...@cloudops.com>
>     >> >>> > wrote:
>     >> >>> >
>     >> >>> > > We haven't opened a ticket yet because we don't have a
> strategy
>     >> yet.
>     >> >>> > >
>     >> >>> > > What do you guys think of this:
>     >> >>> > > - We setup a new github repo in the 'apache' org which
> consists
>     >> >>> > > of a
>     >> >>> > single
>     >> >>> > > file with a list of active/supported mirrors.
>     >> >>> > > - I write a small web server, distributed as a binary,
> which can
>     >> >>> > > be
>     >> >>> > hosted
>     >> >>> > > by ASF Infra.  This web server will query the current list
> of
>     >> >>> > > mirrors
>     >> >>> and
>     >> >>> > > will select one and then do a 302 redirect to that mirror.
>     >> >>> > >
>     >> >>> > > The act of 'choosing' a mirror could be done in a number of
> ways.
>     >> >>> > > - If we want to define an order, then it could just try
> from the
>     >> >>> > > top of
>     >> >>> > the
>     >> >>> > > list and work its way down.  It would curl the target to
> make
>     >> >>> > > sure it
>     >> >>> > gets
>     >> >>> > > a 200 and if it does, it would do a 302 redirect.
>     >> >>> > > - Or, if we want to distribute the load across the mirrors,
> we
>     >> >>> > > could
>     >> >>> pick
>     >> >>> > > from the list randomly.  Again, doing a curl to verify the
> mirror
>     >> >>> > > is up
>     >> >>> > and
>     >> >>> > > then doing a redirect.
>     >> >>> > > - If we want to get fancy, we could do a reverse IP lookup
> and
>     >> >>> > > try to
>     >> >>> > match
>     >> >>> > > the requester with their closest geographical mirror.
>     >> >>> > >
>     >> >>> > > Thoughts?
>     >> >>> > >
>     >> >>> > > *Will STEVENS*
>     >> >>> > > Lead Developer
>     >> >>> > >
>     >> >>> > > <https://goo.gl/NYZ8KK>
>     >> >>> > >
>     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
>     >> >>> chiradeepv@gmail.com
>     >> >>> > >
>     >> >>> > > wrote:
>     >> >>> > >
>     >> >>> > > > What steps are needed to set up a mirror? What does Infra
> need
>     >> to do?
>     >> >>> > Has
>     >> >>> > > > anybody filed a ticket with Infra?
>     >> >>> > > >
>     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
>     >> >>> > > > raja.pullela@accelerite.com>
>     >> >>> > > > wrote:
>     >> >>> > > >
>     >> >>> > > > > Hi will,
>     >> >>> > > > >
>     >> >>> > > > > I believe, we didn’t get to close ‘getting a mirror on
> Apache’
>     >> >>> > because
>     >> >>> > > we
>     >> >>> > > > > needed someone on the Apache Infra side to close this.
> BTW,
>     >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
>     >> >>> > > > > all/most of
>     >> >>> > the
>     >> >>> > > > > content.  Once we can close on the Apache mirror for
> hosting
>     >> >>> > > > > the
>     >> >>> > > > content, I
>     >> >>> > > > > can help assist getting the content there.
>     >> >>> > > > >
>     >> >>> > > > > For now, we have replicated the download.cloud.com
> content to
>     >> >>> > > > > ‘ s3.download.accelerite.com’.
>     >> >>> > > > > Also, we are working on a set of steps/procedure to
> help with
>     >> >>> > > > > this change.  I will update everyone in about a week’s
> time
>     >> >>> > > > > on the
>     >> >>> > details.
>     >> >>> > > > >
>     >> >>> > > > > Best,
>     >> >>> > > > > Raja Pullela
>     >> >>> > > > > Engineering Team,
>     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA,
95054
>     >> >>> > > > >
>     >> >>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on
> behalf of
>     >> >>> > > > > Will Stevens" <williamstevens@gmail.com on behalf of
>     >> >>> > wstevens@cloudops.com>
>     >> >>> > > > > wrote:
>     >> >>> > > > >
>     >> >>> > > > > unfortunately the template mirror conversation got
> caught up
>     >> >>> > > > > in
>     >> >>> > details
>     >> >>> > > > and
>     >> >>> > > > > nobody took the lead on implementing a solution.
>     >> >>> > > > >
>     >> >>> > > > > citrix has been pinging me every couple months to say
> 'dude,
>     >> >>> > > > > we
>     >> >>> need
>     >> >>> > to
>     >> >>> > > > > remove the dependency on download.citrix.com', but i
> have not
>     >> >>> > > > > had
>     >> >>> > the
>     >> >>> > > > > cycles to get in and solve the problem.  the shutdown
> of that
>     >> >>> > > > > is
>     >> >>> > > imminent
>     >> >>> > > > > right now, so we need to solve it asap.
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > > *Will STEVENS*
>     >> >>> > > > > Lead Developer
>     >> >>> > > > >
>     >> >>> > > > > <https://goo.gl/NYZ8KK>
>     >> >>> > > > >
>     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
>     >> >>> > paul.angus@shapeblue.com
>     >> >>> > > >
>     >> >>> > > > > wrote:
>     >> >>> > > > >
>     >> >>> > > > > > Hi Nathan,
>     >> >>> > > > > >
>     >> >>> > > > > > Ideally, if you put the template location in (or use
a
>     >> >>> > > > > > template
>     >> >>> > > defined
>     >> >>> > > > > > in)  test_data.py then the actual location can be
>     >> >>> > > > > > overridden by
>     >> >>> > > anyone
>     >> >>> > > > > > testing.
>     >> >>> > > > > >
>     >> >>> > > > > > For Trillian, we've copied all of the templates that
> people
>     >> >>> > > > > > have
>     >> >>> > > define
>     >> >>> > > > > to
>     >> >>> > > > > > a local repo and then replace the URLs in
> test_data.py to
>     >> >>> > > > > > reduce
>     >> >>> > > > > bandwidth
>     >> >>> > > > > > use and download times.
>     >> >>> > > > > >
>     >> >>> > > > > > Ie:
>     >> >>> > > > > >
>     >> >>> > > > > >             "bootableIso":
>     >> >>> > > > > >                 {
>     >> >>> > > > > >                     "displaytext": "Test Bootable
> ISO",
>     >> >>> > > > > >                     "name": "testISO",
>     >> >>> > > > > >                     "bootable": True,
>     >> >>> > > > > >                     "ispublic": False,
>     >> >>> > > > > >                     "url": "{{ marvin_images_location
>     >> >>> > > > > > }}/TinyCore-current.iso",
>     >> >>> > > > > >                     "ostype": 'Other Linux (64-bit)',
>     >> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
>     >> >>> > > > > >         },
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > I thought that we had come up with a solution for
>     >> >>> > download.cloud.com
>     >> >>> > > ,
>     >> >>> > > > by
>     >> >>> > > > > > having a mirrorlist hosted in Community Apache
> 'space' with
>     >> >>> anyone
>     >> >>> > > able
>     >> >>> > > > > to
>     >> >>> > > > > > out themselves forward as a mirror.
>     >> >>> > > > > > But I must admit I lost track of whether anyone made
> the
>     >> >>> requisite
>     >> >>> > > > > changes
>     >> >>> > > > > > in code....
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > Kind regards,
>     >> >>> > > > > >
>     >> >>> > > > > > Paul Angus
>     >> >>> > > > > >
>     >> >>> > > > > > paul.angus@shapeblue.com
>     >> >>> > > > > > www.shapeblue.com
>     >> >>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     >> >>> > > > > > @shapeblue
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > -----Original Message-----
>     >> >>> > > > > > From: williamstevens@gmail.com
>     >> >>> > > > > > [mailto:williamstevens@gmail.com]
>     >> >>> > On
>     >> >>> > > > > > Behalf Of Will Stevens
>     >> >>> > > > > > Sent: 24 February 2017 16:30
>     >> >>> > > > > > To: dev@cloudstack.apache.org
>     >> >>> > > > > > Subject: Re: Modern template hosting
>     >> >>> > > > > >
>     >> >>> > > > > > this is a hard questions.  in general, we should be
> setting
>     >> >>> > > > > > up a
>     >> >>> > > mirror
>     >> >>> > > > > on
>     >> >>> > > > > > some cloudstack/apache domain and then mirror to
other
>     >> >>> > > > > > provided
>     >> >>> > > > > templates.
>     >> >>> > > > > >
>     >> >>> > > > > > we MUST come up with a solution to deprecate
>     >> >>> > > > > > 'download.cloud.com
>     >> >>> ',
>     >> >>> > > > that
>     >> >>> > > > > > is going to be going away any day now.
>     >> >>> > > > > >
>     >> >>> > > > > > i don't know the right way to solve this to be
> honest, but
>     >> >>> > > > > > if you
>     >> >>> > > have
>     >> >>> > > > > > ideas, i am willing to help.
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > *Will STEVENS*
>     >> >>> > > > > > Lead Developer
>     >> >>> > > > > >
>     >> >>> > > > > > <https://goo.gl/NYZ8KK>
>     >> >>> > > > > >
>     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
>     >> >>> njohnson@ena.com
>     >> >>> > >
>     >> >>> > > > > wrote:
>     >> >>> > > > > >
>     >> >>> > > > > > > So not to re-open a can of worms, but I’m in a
> situation
>     >> >>> > > > > > > where
>     >> >>> I
>     >> >>> > > need
>     >> >>> > > > > > > to come up with a Marvin component test that
> depends on a
>     >> >>> > template
>     >> >>> > > > > > > based on a kernel that’s relatively new, i.e.,
> newer than
>     >> >>> Centos
>     >> >>> > > 5.3
>     >> >>> > > > /
>     >> >>> > > > > > Ubuntu 10.04 .
>     >> >>> > > > > > > I see openvm.eu has a suitable template (Ubuntu
> 16.0.4
>     >> >>> > > > > > > for
>     >> >>> KVM),
>     >> >>> > > but
>     >> >>> > > > > > > from looking at the thread "Migrating CloudStack
> content
>     >> >>> > > > > > > from download.cloud.com” it looks like there is
>     >> >>> > > > > > > resistance to using
>     >> >>> > > this
>     >> >>> > > > at
>     >> >>> > > > > > > least for hosting system vm templates over concerns
> of
>     >> >>> > neutrality.
>     >> >>> > > > > > > Would this be suitable for a component test?  If
> not,
>     >> >>> > > > > > > what is a
>     >> >>> > > > > “blessed”
>     >> >>> > > > > > > template location?
>     >> >>> > > > > > >
>     >> >>> > > > > > > Thanks in advance!
>     >> >>> > > > > > >
>     >> >>> > > > > > > Nathan Johnson
>     >> >>> > > > > > > R&D Engineer
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville, TN
> 37211
>     >> >>> > > > > > > General Office: 615-312-6000
>     >> >>> > > > > > >
>     >> >>> > > > > > > website | blog | support
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > > DISCLAIMER
>     >> >>> > > > > ==========
>     >> >>> > > > > This e-mail may contain privileged and confidential
>     >> >>> > > > > information
>     >> >>> which
>     >> >>> > > is
>     >> >>> > > > > the property of Accelerite, a Persistent Systems
> business. It
>     >> >>> > > > > is
>     >> >>> > > intended
>     >> >>> > > > > only for the use of the individual or entity to which
> it is
>     >> >>> > addressed.
>     >> >>> > > If
>     >> >>> > > > > you are not the intended recipient, you are not
> authorized to
>     >> >>> > > > > read,
>     >> >>> > > > retain,
>     >> >>> > > > > copy, print, distribute or use this message. If you
have
>     >> >>> > > > > received
>     >> >>> > this
>     >> >>> > > > > communication in error, please notify the sender and
> delete
>     >> >>> > > > > all
>     >> >>> > copies
>     >> >>> > > of
>     >> >>> > > > > this message. Accelerite, a Persistent Systems business
> does
>     >> >>> > > > > not
>     >> >>> > accept
>     >> >>> > > > any
>     >> >>> > > > > liability for virus infected mails.
>     >> >>> > > > >
>     >> >>> > > >
>     >> >>> > >
>     >> >>> >
>     >> >>>
>     >> >
>     >> >abhinandan.prateek@shapeblue.com
>     >> >www.shapeblue.com
>     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>     >> >
>     >> >
>     >> >
>     >>
>     >> abhinandan.prateek@shapeblue.com
>     >> www.shapeblue.com
>     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     >> @shapeblue
>     >>
>     >>
>     >>
>     >>
>     >
>
>
>
> daan.hoogland@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>


--
Rafael Weingärtner

Re: Modern template hosting

Posted by Rafael Weingärtner <ra...@gmail.com>.

Will great job.

I had the same doubt as Daan.

Have you considered using HTTP 300 working mode? Then, we could let the
client decide which mirror is the best(closest?). Section “10.3.1” of [1]
talks about it; this would put the pressure on deciding upon a mirror on
the client side, but I think it would be fair.

[1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

PS: You really like very short variables names!

On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <da...@shapeblue.com>
wrote:

> Nice little thing Will,
>
> One question: if I read the code correctly it ‘go’es and tries all mirrors
> at once and whichever responses first is redirected to the client. This
> might well be the same every time. This might be the one closest to the
> mirror302 and not to the client. Is that correct or did I miss a weighing
> algorithm hidden in there?
>
> Good coding,
>
> On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
>
>     Hey All,
>     Please review this repo and tell me what you think:
>     https://github.com/swill/mirror302
>
>     Let me know if you have questions or you would like me to make changes
>     before I bring this topic up with ASF Infra and open a ticket to get
> this
>     implemented.
>
>     Cheers,
>
>     *Will Stevens*
>
>
>
>     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <ws...@cloudops.com>
> wrote:
>
>     > I am building a short term solution right now.  I hopefully will
> have the
>     > building blocks in place by the weekend so we can start working with
> Infra
>     > to get it in place.
>     >
>     > We will have a web server which we can point something like
>     > downloads.cloudstack.org (or whatever url) at.  Then we will have
> an ASF
>     > repo which tracks a mirror list and exposes a static site to be able
> to
>     > query the mirror list.
>     >
>     > The web server will receive a request and will do a 302 redirect to
> the
>     > appropriate resource on one of the mirrors.
>     >
>     > This gets us started.  Legacy environments will have to do a DB
> change to
>     > change from 'download.cloud.com' to 'downloads.cloudstack.org' (or
>     > whatever), but otherwise it should be pretty simple.
>     >
>     > We can improve how we deliver templates going forward, but this
> obviously
>     > requires some discussion still.  I will try to buy us some time with
> an
>     > implementation which solves our problems today...
>     >
>     > *Will STEVENS*
>     > Lead Developer
>     >
>     > <https://goo.gl/NYZ8KK>
>     >
>     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
>     > abhinandan.prateek@shapeblue.com> wrote:
>     >
>     >>
>     >> Initial seeding is a manual step and that is not going to change in
> near
>     >> future. A handy list of official places from where these templates
> can be
>     >> downloaded won’t harm this part of the installation. Just allow
> anyone
>     >> credible who follows democratic process to publish their templates
> with
>     >> apache’s blessings.
>     >>
>     >>
>     >> Coming to urls hard coded in db, the popular suggestion is to point
> them
>     >> to mirrors. This looks good but this is going to take some effort
> that has
>     >> not materialised yet. Once this change is made it will be going in
> some
>     >> future release like 4.11; the problem still remains for the
> releases that
>     >> are already out there. In case access to download.cloud.com is
> dropped
>     >> at some point we will have to document the procedure such that
> people know
>     >> how to make these stable releases work. That is where the procedure
>     >> documented here https://shankerbalan.net/blog/
>     >> seed-cloudstack-templates-offline/ should be officially adapted
> with a
>     >> marketplace to pick up templates of choosing. Maybe we can have a
> pre-setup
>     >> script to make choices instead of written wiki. Note that this will
> be
>     >> anyway needed by the older releases.
>     >>
>     >>
>     >>
>     >>
>     >> On 01/03/17, 4:24 PM, "Paul Angus" <pa...@shapeblue.com>
> wrote:
>     >>
>     >> >-1
>     >> >
>     >> >We are trying to get MORE people to use CloudStack.  And raising
> the
>     >> barrier to entry does nothing to help that.
>     >> >
>     >> >
>     >> >Kind regards,
>     >> >
>     >> >Paul Angus
>     >> >
>     >> >paul.angus@shapeblue.com
>     >> >www.shapeblue.com
>     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     >> >@shapeblue
>     >> >
>     >> >
>     >> >
>     >> >
>     >> >-----Original Message-----
>     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@shapeblue.com]
>     >> >Sent: 01 March 2017 09:39
>     >> >To: dev@cloudstack.apache.org
>     >> >Subject: Re: Modern template hosting
>     >> >
>     >> >+1
>     >> >
>     >> >After seeing this discussion running in circles several times, I
> think
>     >> we should at least get started with the simplest option.
>     >> >
>     >> >
>     >> >
>     >> >
>     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <ch...@gmail.com>
> wrote:
>     >> >
>     >> >>The alternative foolproof way is to simply not provide the
> automatic
>     >> >>download.
>     >> >>
>     >> >>Just document the mirror list URL where the templates can be
> found. I
>     >> >>don't know why people reject this option.
>     >> >>
>     >> >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
>     >> >>
>     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
>     >> >><wi...@gmail.com>
>     >> >>wrote:
>     >> >>
>     >> >>> OK. Thanks for the heads up.
>     >> >>>
>     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> chiradeepv@gmail.com>
>     >> wrote:
>     >> >>>
>     >> >>> > Sounds workable. The downloader code in the SSVM won't follow
>     >> >>> > redirects I think.
>     >> >>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d
> 9cd0444
>     >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
>     >> >>> > HttpTemplateDownloader.java#L93
>     >> >>> > https://goo.gl/dSi0r5
>     >> >>> >  Might need to add
>     >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
>     >> >>> >
>     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
>     >> >>> > <ws...@cloudops.com>
>     >> >>> > wrote:
>     >> >>> >
>     >> >>> > > We haven't opened a ticket yet because we don't have a
> strategy
>     >> yet.
>     >> >>> > >
>     >> >>> > > What do you guys think of this:
>     >> >>> > > - We setup a new github repo in the 'apache' org which
> consists
>     >> >>> > > of a
>     >> >>> > single
>     >> >>> > > file with a list of active/supported mirrors.
>     >> >>> > > - I write a small web server, distributed as a binary,
> which can
>     >> >>> > > be
>     >> >>> > hosted
>     >> >>> > > by ASF Infra.  This web server will query the current list
> of
>     >> >>> > > mirrors
>     >> >>> and
>     >> >>> > > will select one and then do a 302 redirect to that mirror.
>     >> >>> > >
>     >> >>> > > The act of 'choosing' a mirror could be done in a number of
> ways.
>     >> >>> > > - If we want to define an order, then it could just try
> from the
>     >> >>> > > top of
>     >> >>> > the
>     >> >>> > > list and work its way down.  It would curl the target to
> make
>     >> >>> > > sure it
>     >> >>> > gets
>     >> >>> > > a 200 and if it does, it would do a 302 redirect.
>     >> >>> > > - Or, if we want to distribute the load across the mirrors,
> we
>     >> >>> > > could
>     >> >>> pick
>     >> >>> > > from the list randomly.  Again, doing a curl to verify the
> mirror
>     >> >>> > > is up
>     >> >>> > and
>     >> >>> > > then doing a redirect.
>     >> >>> > > - If we want to get fancy, we could do a reverse IP lookup
> and
>     >> >>> > > try to
>     >> >>> > match
>     >> >>> > > the requester with their closest geographical mirror.
>     >> >>> > >
>     >> >>> > > Thoughts?
>     >> >>> > >
>     >> >>> > > *Will STEVENS*
>     >> >>> > > Lead Developer
>     >> >>> > >
>     >> >>> > > <https://goo.gl/NYZ8KK>
>     >> >>> > >
>     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
>     >> >>> chiradeepv@gmail.com
>     >> >>> > >
>     >> >>> > > wrote:
>     >> >>> > >
>     >> >>> > > > What steps are needed to set up a mirror? What does Infra
> need
>     >> to do?
>     >> >>> > Has
>     >> >>> > > > anybody filed a ticket with Infra?
>     >> >>> > > >
>     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
>     >> >>> > > > raja.pullela@accelerite.com>
>     >> >>> > > > wrote:
>     >> >>> > > >
>     >> >>> > > > > Hi will,
>     >> >>> > > > >
>     >> >>> > > > > I believe, we didn’t get to close ‘getting a mirror on
> Apache’
>     >> >>> > because
>     >> >>> > > we
>     >> >>> > > > > needed someone on the Apache Infra side to close this.
> BTW,
>     >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
>     >> >>> > > > > all/most of
>     >> >>> > the
>     >> >>> > > > > content.  Once we can close on the Apache mirror for
> hosting
>     >> >>> > > > > the
>     >> >>> > > > content, I
>     >> >>> > > > > can help assist getting the content there.
>     >> >>> > > > >
>     >> >>> > > > > For now, we have replicated the download.cloud.com
> content to
>     >> >>> > > > > ‘ s3.download.accelerite.com’.
>     >> >>> > > > > Also, we are working on a set of steps/procedure to
> help with
>     >> >>> > > > > this change.  I will update everyone in about a week’s
> time
>     >> >>> > > > > on the
>     >> >>> > details.
>     >> >>> > > > >
>     >> >>> > > > > Best,
>     >> >>> > > > > Raja Pullela
>     >> >>> > > > > Engineering Team,
>     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA, 95054
>     >> >>> > > > >
>     >> >>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on
> behalf of
>     >> >>> > > > > Will Stevens" <williamstevens@gmail.com on behalf of
>     >> >>> > wstevens@cloudops.com>
>     >> >>> > > > > wrote:
>     >> >>> > > > >
>     >> >>> > > > > unfortunately the template mirror conversation got
> caught up
>     >> >>> > > > > in
>     >> >>> > details
>     >> >>> > > > and
>     >> >>> > > > > nobody took the lead on implementing a solution.
>     >> >>> > > > >
>     >> >>> > > > > citrix has been pinging me every couple months to say
> 'dude,
>     >> >>> > > > > we
>     >> >>> need
>     >> >>> > to
>     >> >>> > > > > remove the dependency on download.citrix.com', but i
> have not
>     >> >>> > > > > had
>     >> >>> > the
>     >> >>> > > > > cycles to get in and solve the problem.  the shutdown
> of that
>     >> >>> > > > > is
>     >> >>> > > imminent
>     >> >>> > > > > right now, so we need to solve it asap.
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > > *Will STEVENS*
>     >> >>> > > > > Lead Developer
>     >> >>> > > > >
>     >> >>> > > > > <https://goo.gl/NYZ8KK>
>     >> >>> > > > >
>     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
>     >> >>> > paul.angus@shapeblue.com
>     >> >>> > > >
>     >> >>> > > > > wrote:
>     >> >>> > > > >
>     >> >>> > > > > > Hi Nathan,
>     >> >>> > > > > >
>     >> >>> > > > > > Ideally, if you put the template location in (or use a
>     >> >>> > > > > > template
>     >> >>> > > defined
>     >> >>> > > > > > in)  test_data.py then the actual location can be
>     >> >>> > > > > > overridden by
>     >> >>> > > anyone
>     >> >>> > > > > > testing.
>     >> >>> > > > > >
>     >> >>> > > > > > For Trillian, we've copied all of the templates that
> people
>     >> >>> > > > > > have
>     >> >>> > > define
>     >> >>> > > > > to
>     >> >>> > > > > > a local repo and then replace the URLs in
> test_data.py to
>     >> >>> > > > > > reduce
>     >> >>> > > > > bandwidth
>     >> >>> > > > > > use and download times.
>     >> >>> > > > > >
>     >> >>> > > > > > Ie:
>     >> >>> > > > > >
>     >> >>> > > > > >             "bootableIso":
>     >> >>> > > > > >                 {
>     >> >>> > > > > >                     "displaytext": "Test Bootable
> ISO",
>     >> >>> > > > > >                     "name": "testISO",
>     >> >>> > > > > >                     "bootable": True,
>     >> >>> > > > > >                     "ispublic": False,
>     >> >>> > > > > >                     "url": "{{ marvin_images_location
>     >> >>> > > > > > }}/TinyCore-current.iso",
>     >> >>> > > > > >                     "ostype": 'Other Linux (64-bit)',
>     >> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
>     >> >>> > > > > >         },
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > I thought that we had come up with a solution for
>     >> >>> > download.cloud.com
>     >> >>> > > ,
>     >> >>> > > > by
>     >> >>> > > > > > having a mirrorlist hosted in Community Apache
> 'space' with
>     >> >>> anyone
>     >> >>> > > able
>     >> >>> > > > > to
>     >> >>> > > > > > out themselves forward as a mirror.
>     >> >>> > > > > > But I must admit I lost track of whether anyone made
> the
>     >> >>> requisite
>     >> >>> > > > > changes
>     >> >>> > > > > > in code....
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > Kind regards,
>     >> >>> > > > > >
>     >> >>> > > > > > Paul Angus
>     >> >>> > > > > >
>     >> >>> > > > > > paul.angus@shapeblue.com
>     >> >>> > > > > > www.shapeblue.com
>     >> >>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     >> >>> > > > > > @shapeblue
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > -----Original Message-----
>     >> >>> > > > > > From: williamstevens@gmail.com
>     >> >>> > > > > > [mailto:williamstevens@gmail.com]
>     >> >>> > On
>     >> >>> > > > > > Behalf Of Will Stevens
>     >> >>> > > > > > Sent: 24 February 2017 16:30
>     >> >>> > > > > > To: dev@cloudstack.apache.org
>     >> >>> > > > > > Subject: Re: Modern template hosting
>     >> >>> > > > > >
>     >> >>> > > > > > this is a hard questions.  in general, we should be
> setting
>     >> >>> > > > > > up a
>     >> >>> > > mirror
>     >> >>> > > > > on
>     >> >>> > > > > > some cloudstack/apache domain and then mirror to other
>     >> >>> > > > > > provided
>     >> >>> > > > > templates.
>     >> >>> > > > > >
>     >> >>> > > > > > we MUST come up with a solution to deprecate
>     >> >>> > > > > > 'download.cloud.com
>     >> >>> ',
>     >> >>> > > > that
>     >> >>> > > > > > is going to be going away any day now.
>     >> >>> > > > > >
>     >> >>> > > > > > i don't know the right way to solve this to be
> honest, but
>     >> >>> > > > > > if you
>     >> >>> > > have
>     >> >>> > > > > > ideas, i am willing to help.
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > *Will STEVENS*
>     >> >>> > > > > > Lead Developer
>     >> >>> > > > > >
>     >> >>> > > > > > <https://goo.gl/NYZ8KK>
>     >> >>> > > > > >
>     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
>     >> >>> njohnson@ena.com
>     >> >>> > >
>     >> >>> > > > > wrote:
>     >> >>> > > > > >
>     >> >>> > > > > > > So not to re-open a can of worms, but I’m in a
> situation
>     >> >>> > > > > > > where
>     >> >>> I
>     >> >>> > > need
>     >> >>> > > > > > > to come up with a Marvin component test that
> depends on a
>     >> >>> > template
>     >> >>> > > > > > > based on a kernel that’s relatively new, i.e.,
> newer than
>     >> >>> Centos
>     >> >>> > > 5.3
>     >> >>> > > > /
>     >> >>> > > > > > Ubuntu 10.04 .
>     >> >>> > > > > > > I see openvm.eu has a suitable template (Ubuntu
> 16.0.4
>     >> >>> > > > > > > for
>     >> >>> KVM),
>     >> >>> > > but
>     >> >>> > > > > > > from looking at the thread "Migrating CloudStack
> content
>     >> >>> > > > > > > from download.cloud.com” it looks like there is
>     >> >>> > > > > > > resistance to using
>     >> >>> > > this
>     >> >>> > > > at
>     >> >>> > > > > > > least for hosting system vm templates over concerns
> of
>     >> >>> > neutrality.
>     >> >>> > > > > > > Would this be suitable for a component test?  If
> not,
>     >> >>> > > > > > > what is a
>     >> >>> > > > > “blessed”
>     >> >>> > > > > > > template location?
>     >> >>> > > > > > >
>     >> >>> > > > > > > Thanks in advance!
>     >> >>> > > > > > >
>     >> >>> > > > > > > Nathan Johnson
>     >> >>> > > > > > > R&D Engineer
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville, TN
> 37211
>     >> >>> > > > > > > General Office: 615-312-6000
>     >> >>> > > > > > >
>     >> >>> > > > > > > website | blog | support
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > > DISCLAIMER
>     >> >>> > > > > ==========
>     >> >>> > > > > This e-mail may contain privileged and confidential
>     >> >>> > > > > information
>     >> >>> which
>     >> >>> > > is
>     >> >>> > > > > the property of Accelerite, a Persistent Systems
> business. It
>     >> >>> > > > > is
>     >> >>> > > intended
>     >> >>> > > > > only for the use of the individual or entity to which
> it is
>     >> >>> > addressed.
>     >> >>> > > If
>     >> >>> > > > > you are not the intended recipient, you are not
> authorized to
>     >> >>> > > > > read,
>     >> >>> > > > retain,
>     >> >>> > > > > copy, print, distribute or use this message. If you have
>     >> >>> > > > > received
>     >> >>> > this
>     >> >>> > > > > communication in error, please notify the sender and
> delete
>     >> >>> > > > > all
>     >> >>> > copies
>     >> >>> > > of
>     >> >>> > > > > this message. Accelerite, a Persistent Systems business
> does
>     >> >>> > > > > not
>     >> >>> > accept
>     >> >>> > > > any
>     >> >>> > > > > liability for virus infected mails.
>     >> >>> > > > >
>     >> >>> > > >
>     >> >>> > >
>     >> >>> >
>     >> >>>
>     >> >
>     >> >abhinandan.prateek@shapeblue.com
>     >> >www.shapeblue.com
>     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>     >> >
>     >> >
>     >> >
>     >>
>     >> abhinandan.prateek@shapeblue.com
>     >> www.shapeblue.com
>     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     >> @shapeblue
>     >>
>     >>
>     >>
>     >>
>     >
>
>
>
> daan.hoogland@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>


-- 
Rafael Weingärtner

Re: Modern template hosting

Posted by Will Stevens <wi...@gmail.com>.

You understand the logic correctly.

Yes, it will favor mirrors which are closest to the hosting of mirror302.
However, if there is a problem with a mirror being overloaded, another
mirror will win the race and serve the resource.

I originally started writing it to pick the closest mirrors geographically
and then verify with head that the selected mirror has the requested
resource.
With this approach I was getting the feeling that I was over complicating
the problem. How do I verify if the closest mirror is having problems.
Let's say it is over loaded or has some other problem and takes twice as
long to deliver the resource as a different mirror. How would I be able to
verify this and be sure the resource can be served in a timely fashion?

The geo way is very deterministic about which mirror is chosen. That is
good for most cases, but if a mirror is having a hard time but is still
responding to head, there does not seem to be a good way for the impacted
client to delivered a different mirror.

I released the code with this simplified implementation for a few reasons.

1) it is easier to understand and is more likely to handle the case that a
mirror is struggling and not direct clients to it.
2) verify with the community that this approach to solving the problem is
acceptable before spending a lot of time to 'perfect' it.
3) get feedback on the issues around geo selection before trying to do a
final implementation.
4) have something to show in order to open a ticket with ASF infra to start
working through the process of getting this in place.

One other note around where the mirror list is hosted. Maybe we can serve
the mirror list from cloudstack-www. We already have automatic publishing
of the static resources on commit to the ASF. Is there a reason we need
anything more complicated than that?

Anyway, would love feedback all around. I am happy to adapt this as we see
fit, but want to make sure we are generally happy with the approach before
I invest a bunch of time into it.

Thanks for the review. :)

On Mar 3, 2017 1:24 AM, "Daan Hoogland" <da...@shapeblue.com> wrote:

> Nice little thing Will,
>
> One question: if I read the code correctly it ‘go’es and tries all mirrors
> at once and whichever responses first is redirected to the client. This
> might well be the same every time. This might be the one closest to the
> mirror302 and not to the client. Is that correct or did I miss a weighing
> algorithm hidden in there?
>
> Good coding,
>
> On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
>
>     Hey All,
>     Please review this repo and tell me what you think:
>     https://github.com/swill/mirror302
>
>     Let me know if you have questions or you would like me to make changes
>     before I bring this topic up with ASF Infra and open a ticket to get
> this
>     implemented.
>
>     Cheers,
>
>     *Will Stevens*
>
>
>
>     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <ws...@cloudops.com>
> wrote:
>
>     > I am building a short term solution right now.  I hopefully will
> have the
>     > building blocks in place by the weekend so we can start working with
> Infra
>     > to get it in place.
>     >
>     > We will have a web server which we can point something like
>     > downloads.cloudstack.org (or whatever url) at.  Then we will have
> an ASF
>     > repo which tracks a mirror list and exposes a static site to be able
> to
>     > query the mirror list.
>     >
>     > The web server will receive a request and will do a 302 redirect to
> the
>     > appropriate resource on one of the mirrors.
>     >
>     > This gets us started.  Legacy environments will have to do a DB
> change to
>     > change from 'download.cloud.com' to 'downloads.cloudstack.org' (or
>     > whatever), but otherwise it should be pretty simple.
>     >
>     > We can improve how we deliver templates going forward, but this
> obviously
>     > requires some discussion still.  I will try to buy us some time with
> an
>     > implementation which solves our problems today...
>     >
>     > *Will STEVENS*
>     > Lead Developer
>     >
>     > <https://goo.gl/NYZ8KK>
>     >
>     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
>     > abhinandan.prateek@shapeblue.com> wrote:
>     >
>     >>
>     >> Initial seeding is a manual step and that is not going to change in
> near
>     >> future. A handy list of official places from where these templates
> can be
>     >> downloaded won’t harm this part of the installation. Just allow
> anyone
>     >> credible who follows democratic process to publish their templates
> with
>     >> apache’s blessings.
>     >>
>     >>
>     >> Coming to urls hard coded in db, the popular suggestion is to point
> them
>     >> to mirrors. This looks good but this is going to take some effort
> that has
>     >> not materialised yet. Once this change is made it will be going in
> some
>     >> future release like 4.11; the problem still remains for the
> releases that
>     >> are already out there. In case access to download.cloud.com is
> dropped
>     >> at some point we will have to document the procedure such that
> people know
>     >> how to make these stable releases work. That is where the procedure
>     >> documented here https://shankerbalan.net/blog/
>     >> seed-cloudstack-templates-offline/ should be officially adapted
> with a
>     >> marketplace to pick up templates of choosing. Maybe we can have a
> pre-setup
>     >> script to make choices instead of written wiki. Note that this will
> be
>     >> anyway needed by the older releases.
>     >>
>     >>
>     >>
>     >>
>     >> On 01/03/17, 4:24 PM, "Paul Angus" <pa...@shapeblue.com>
> wrote:
>     >>
>     >> >-1
>     >> >
>     >> >We are trying to get MORE people to use CloudStack.  And raising
> the
>     >> barrier to entry does nothing to help that.
>     >> >
>     >> >
>     >> >Kind regards,
>     >> >
>     >> >Paul Angus
>     >> >
>     >> >paul.angus@shapeblue.com
>     >> >www.shapeblue.com
>     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     >> >@shapeblue
>     >> >
>     >> >
>     >> >
>     >> >
>     >> >-----Original Message-----
>     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@shapeblue.com]
>     >> >Sent: 01 March 2017 09:39
>     >> >To: dev@cloudstack.apache.org
>     >> >Subject: Re: Modern template hosting
>     >> >
>     >> >+1
>     >> >
>     >> >After seeing this discussion running in circles several times, I
> think
>     >> we should at least get started with the simplest option.
>     >> >
>     >> >
>     >> >
>     >> >
>     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <ch...@gmail.com>
> wrote:
>     >> >
>     >> >>The alternative foolproof way is to simply not provide the
> automatic
>     >> >>download.
>     >> >>
>     >> >>Just document the mirror list URL where the templates can be
> found. I
>     >> >>don't know why people reject this option.
>     >> >>
>     >> >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
>     >> >>
>     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
>     >> >><wi...@gmail.com>
>     >> >>wrote:
>     >> >>
>     >> >>> OK. Thanks for the heads up.
>     >> >>>
>     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> chiradeepv@gmail.com>
>     >> wrote:
>     >> >>>
>     >> >>> > Sounds workable. The downloader code in the SSVM won't follow
>     >> >>> > redirects I think.
>     >> >>> > https://github.com/apache/cloudstack/blob/
> 5511065fc20787619d9cd0444
>     >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
>     >> >>> > HttpTemplateDownloader.java#L93
>     >> >>> > https://goo.gl/dSi0r5
>     >> >>> >  Might need to add
>     >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
>     >> >>> >
>     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
>     >> >>> > <ws...@cloudops.com>
>     >> >>> > wrote:
>     >> >>> >
>     >> >>> > > We haven't opened a ticket yet because we don't have a
> strategy
>     >> yet.
>     >> >>> > >
>     >> >>> > > What do you guys think of this:
>     >> >>> > > - We setup a new github repo in the 'apache' org which
> consists
>     >> >>> > > of a
>     >> >>> > single
>     >> >>> > > file with a list of active/supported mirrors.
>     >> >>> > > - I write a small web server, distributed as a binary,
> which can
>     >> >>> > > be
>     >> >>> > hosted
>     >> >>> > > by ASF Infra.  This web server will query the current list
> of
>     >> >>> > > mirrors
>     >> >>> and
>     >> >>> > > will select one and then do a 302 redirect to that mirror.
>     >> >>> > >
>     >> >>> > > The act of 'choosing' a mirror could be done in a number of
> ways.
>     >> >>> > > - If we want to define an order, then it could just try
> from the
>     >> >>> > > top of
>     >> >>> > the
>     >> >>> > > list and work its way down.  It would curl the target to
> make
>     >> >>> > > sure it
>     >> >>> > gets
>     >> >>> > > a 200 and if it does, it would do a 302 redirect.
>     >> >>> > > - Or, if we want to distribute the load across the mirrors,
> we
>     >> >>> > > could
>     >> >>> pick
>     >> >>> > > from the list randomly.  Again, doing a curl to verify the
> mirror
>     >> >>> > > is up
>     >> >>> > and
>     >> >>> > > then doing a redirect.
>     >> >>> > > - If we want to get fancy, we could do a reverse IP lookup
> and
>     >> >>> > > try to
>     >> >>> > match
>     >> >>> > > the requester with their closest geographical mirror.
>     >> >>> > >
>     >> >>> > > Thoughts?
>     >> >>> > >
>     >> >>> > > *Will STEVENS*
>     >> >>> > > Lead Developer
>     >> >>> > >
>     >> >>> > > <https://goo.gl/NYZ8KK>
>     >> >>> > >
>     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
>     >> >>> chiradeepv@gmail.com
>     >> >>> > >
>     >> >>> > > wrote:
>     >> >>> > >
>     >> >>> > > > What steps are needed to set up a mirror? What does Infra
> need
>     >> to do?
>     >> >>> > Has
>     >> >>> > > > anybody filed a ticket with Infra?
>     >> >>> > > >
>     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
>     >> >>> > > > raja.pullela@accelerite.com>
>     >> >>> > > > wrote:
>     >> >>> > > >
>     >> >>> > > > > Hi will,
>     >> >>> > > > >
>     >> >>> > > > > I believe, we didn’t get to close ‘getting a mirror on
> Apache’
>     >> >>> > because
>     >> >>> > > we
>     >> >>> > > > > needed someone on the Apache Infra side to close this.
> BTW,
>     >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
>     >> >>> > > > > all/most of
>     >> >>> > the
>     >> >>> > > > > content.  Once we can close on the Apache mirror for
> hosting
>     >> >>> > > > > the
>     >> >>> > > > content, I
>     >> >>> > > > > can help assist getting the content there.
>     >> >>> > > > >
>     >> >>> > > > > For now, we have replicated the download.cloud.com
> content to
>     >> >>> > > > > ‘ s3.download.accelerite.com’.
>     >> >>> > > > > Also, we are working on a set of steps/procedure to
> help with
>     >> >>> > > > > this change.  I will update everyone in about a week’s
> time
>     >> >>> > > > > on the
>     >> >>> > details.
>     >> >>> > > > >
>     >> >>> > > > > Best,
>     >> >>> > > > > Raja Pullela
>     >> >>> > > > > Engineering Team,
>     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA, 95054
>     >> >>> > > > >
>     >> >>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on
> behalf of
>     >> >>> > > > > Will Stevens" <williamstevens@gmail.com on behalf of
>     >> >>> > wstevens@cloudops.com>
>     >> >>> > > > > wrote:
>     >> >>> > > > >
>     >> >>> > > > > unfortunately the template mirror conversation got
> caught up
>     >> >>> > > > > in
>     >> >>> > details
>     >> >>> > > > and
>     >> >>> > > > > nobody took the lead on implementing a solution.
>     >> >>> > > > >
>     >> >>> > > > > citrix has been pinging me every couple months to say
> 'dude,
>     >> >>> > > > > we
>     >> >>> need
>     >> >>> > to
>     >> >>> > > > > remove the dependency on download.citrix.com', but i
> have not
>     >> >>> > > > > had
>     >> >>> > the
>     >> >>> > > > > cycles to get in and solve the problem.  the shutdown
> of that
>     >> >>> > > > > is
>     >> >>> > > imminent
>     >> >>> > > > > right now, so we need to solve it asap.
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > > *Will STEVENS*
>     >> >>> > > > > Lead Developer
>     >> >>> > > > >
>     >> >>> > > > > <https://goo.gl/NYZ8KK>
>     >> >>> > > > >
>     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
>     >> >>> > paul.angus@shapeblue.com
>     >> >>> > > >
>     >> >>> > > > > wrote:
>     >> >>> > > > >
>     >> >>> > > > > > Hi Nathan,
>     >> >>> > > > > >
>     >> >>> > > > > > Ideally, if you put the template location in (or use a
>     >> >>> > > > > > template
>     >> >>> > > defined
>     >> >>> > > > > > in)  test_data.py then the actual location can be
>     >> >>> > > > > > overridden by
>     >> >>> > > anyone
>     >> >>> > > > > > testing.
>     >> >>> > > > > >
>     >> >>> > > > > > For Trillian, we've copied all of the templates that
> people
>     >> >>> > > > > > have
>     >> >>> > > define
>     >> >>> > > > > to
>     >> >>> > > > > > a local repo and then replace the URLs in
> test_data.py to
>     >> >>> > > > > > reduce
>     >> >>> > > > > bandwidth
>     >> >>> > > > > > use and download times.
>     >> >>> > > > > >
>     >> >>> > > > > > Ie:
>     >> >>> > > > > >
>     >> >>> > > > > >             "bootableIso":
>     >> >>> > > > > >                 {
>     >> >>> > > > > >                     "displaytext": "Test Bootable
> ISO",
>     >> >>> > > > > >                     "name": "testISO",
>     >> >>> > > > > >                     "bootable": True,
>     >> >>> > > > > >                     "ispublic": False,
>     >> >>> > > > > >                     "url": "{{ marvin_images_location
>     >> >>> > > > > > }}/TinyCore-current.iso",
>     >> >>> > > > > >                     "ostype": 'Other Linux (64-bit)',
>     >> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
>     >> >>> > > > > >         },
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > I thought that we had come up with a solution for
>     >> >>> > download.cloud.com
>     >> >>> > > ,
>     >> >>> > > > by
>     >> >>> > > > > > having a mirrorlist hosted in Community Apache
> 'space' with
>     >> >>> anyone
>     >> >>> > > able
>     >> >>> > > > > to
>     >> >>> > > > > > out themselves forward as a mirror.
>     >> >>> > > > > > But I must admit I lost track of whether anyone made
> the
>     >> >>> requisite
>     >> >>> > > > > changes
>     >> >>> > > > > > in code....
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > Kind regards,
>     >> >>> > > > > >
>     >> >>> > > > > > Paul Angus
>     >> >>> > > > > >
>     >> >>> > > > > > paul.angus@shapeblue.com
>     >> >>> > > > > > www.shapeblue.com
>     >> >>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     >> >>> > > > > > @shapeblue
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > -----Original Message-----
>     >> >>> > > > > > From: williamstevens@gmail.com
>     >> >>> > > > > > [mailto:williamstevens@gmail.com]
>     >> >>> > On
>     >> >>> > > > > > Behalf Of Will Stevens
>     >> >>> > > > > > Sent: 24 February 2017 16:30
>     >> >>> > > > > > To: dev@cloudstack.apache.org
>     >> >>> > > > > > Subject: Re: Modern template hosting
>     >> >>> > > > > >
>     >> >>> > > > > > this is a hard questions.  in general, we should be
> setting
>     >> >>> > > > > > up a
>     >> >>> > > mirror
>     >> >>> > > > > on
>     >> >>> > > > > > some cloudstack/apache domain and then mirror to other
>     >> >>> > > > > > provided
>     >> >>> > > > > templates.
>     >> >>> > > > > >
>     >> >>> > > > > > we MUST come up with a solution to deprecate
>     >> >>> > > > > > 'download.cloud.com
>     >> >>> ',
>     >> >>> > > > that
>     >> >>> > > > > > is going to be going away any day now.
>     >> >>> > > > > >
>     >> >>> > > > > > i don't know the right way to solve this to be
> honest, but
>     >> >>> > > > > > if you
>     >> >>> > > have
>     >> >>> > > > > > ideas, i am willing to help.
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > *Will STEVENS*
>     >> >>> > > > > > Lead Developer
>     >> >>> > > > > >
>     >> >>> > > > > > <https://goo.gl/NYZ8KK>
>     >> >>> > > > > >
>     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
>     >> >>> njohnson@ena.com
>     >> >>> > >
>     >> >>> > > > > wrote:
>     >> >>> > > > > >
>     >> >>> > > > > > > So not to re-open a can of worms, but I’m in a
> situation
>     >> >>> > > > > > > where
>     >> >>> I
>     >> >>> > > need
>     >> >>> > > > > > > to come up with a Marvin component test that
> depends on a
>     >> >>> > template
>     >> >>> > > > > > > based on a kernel that’s relatively new, i.e.,
> newer than
>     >> >>> Centos
>     >> >>> > > 5.3
>     >> >>> > > > /
>     >> >>> > > > > > Ubuntu 10.04 .
>     >> >>> > > > > > > I see openvm.eu has a suitable template (Ubuntu
> 16.0.4
>     >> >>> > > > > > > for
>     >> >>> KVM),
>     >> >>> > > but
>     >> >>> > > > > > > from looking at the thread "Migrating CloudStack
> content
>     >> >>> > > > > > > from download.cloud.com” it looks like there is
>     >> >>> > > > > > > resistance to using
>     >> >>> > > this
>     >> >>> > > > at
>     >> >>> > > > > > > least for hosting system vm templates over concerns
> of
>     >> >>> > neutrality.
>     >> >>> > > > > > > Would this be suitable for a component test?  If
> not,
>     >> >>> > > > > > > what is a
>     >> >>> > > > > “blessed”
>     >> >>> > > > > > > template location?
>     >> >>> > > > > > >
>     >> >>> > > > > > > Thanks in advance!
>     >> >>> > > > > > >
>     >> >>> > > > > > > Nathan Johnson
>     >> >>> > > > > > > R&D Engineer
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville, TN
> 37211
>     >> >>> > > > > > > General Office: 615-312-6000
>     >> >>> > > > > > >
>     >> >>> > > > > > > website | blog | support
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > > DISCLAIMER
>     >> >>> > > > > ==========
>     >> >>> > > > > This e-mail may contain privileged and confidential
>     >> >>> > > > > information
>     >> >>> which
>     >> >>> > > is
>     >> >>> > > > > the property of Accelerite, a Persistent Systems
> business. It
>     >> >>> > > > > is
>     >> >>> > > intended
>     >> >>> > > > > only for the use of the individual or entity to which
> it is
>     >> >>> > addressed.
>     >> >>> > > If
>     >> >>> > > > > you are not the intended recipient, you are not
> authorized to
>     >> >>> > > > > read,
>     >> >>> > > > retain,
>     >> >>> > > > > copy, print, distribute or use this message. If you have
>     >> >>> > > > > received
>     >> >>> > this
>     >> >>> > > > > communication in error, please notify the sender and
> delete
>     >> >>> > > > > all
>     >> >>> > copies
>     >> >>> > > of
>     >> >>> > > > > this message. Accelerite, a Persistent Systems business
> does
>     >> >>> > > > > not
>     >> >>> > accept
>     >> >>> > > > any
>     >> >>> > > > > liability for virus infected mails.
>     >> >>> > > > >
>     >> >>> > > >
>     >> >>> > >
>     >> >>> >
>     >> >>>
>     >> >
>     >> >abhinandan.prateek@shapeblue.com
>     >> >www.shapeblue.com
>     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>     >> >
>     >> >
>     >> >
>     >>
>     >> abhinandan.prateek@shapeblue.com
>     >> www.shapeblue.com
>     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     >> @shapeblue
>     >>
>     >>
>     >>
>     >>
>     >
>
>
>
> daan.hoogland@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>

Re: Modern template hosting

Posted by Daan Hoogland <da...@shapeblue.com>.

Nice little thing Will,

One question: if I read the code correctly it ‘go’es and tries all mirrors at once and whichever responses first is redirected to the client. This might well be the same every time. This might be the one closest to the mirror302 and not to the client. Is that correct or did I miss a weighing algorithm hidden in there?

Good coding,

On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:

    Hey All,
    Please review this repo and tell me what you think:
    https://github.com/swill/mirror302
    
    Let me know if you have questions or you would like me to make changes
    before I bring this topic up with ASF Infra and open a ticket to get this
    implemented.
    
    Cheers,
    
    *Will Stevens*
    
    
    
    On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <ws...@cloudops.com> wrote:
    
    > I am building a short term solution right now.  I hopefully will have the
    > building blocks in place by the weekend so we can start working with Infra
    > to get it in place.
    >
    > We will have a web server which we can point something like
    > downloads.cloudstack.org (or whatever url) at.  Then we will have an ASF
    > repo which tracks a mirror list and exposes a static site to be able to
    > query the mirror list.
    >
    > The web server will receive a request and will do a 302 redirect to the
    > appropriate resource on one of the mirrors.
    >
    > This gets us started.  Legacy environments will have to do a DB change to
    > change from 'download.cloud.com' to 'downloads.cloudstack.org' (or
    > whatever), but otherwise it should be pretty simple.
    >
    > We can improve how we deliver templates going forward, but this obviously
    > requires some discussion still.  I will try to buy us some time with an
    > implementation which solves our problems today...
    >
    > *Will STEVENS*
    > Lead Developer
    >
    > <https://goo.gl/NYZ8KK>
    >
    > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
    > abhinandan.prateek@shapeblue.com> wrote:
    >
    >>
    >> Initial seeding is a manual step and that is not going to change in near
    >> future. A handy list of official places from where these templates can be
    >> downloaded won’t harm this part of the installation. Just allow anyone
    >> credible who follows democratic process to publish their templates with
    >> apache’s blessings.
    >>
    >>
    >> Coming to urls hard coded in db, the popular suggestion is to point them
    >> to mirrors. This looks good but this is going to take some effort that has
    >> not materialised yet. Once this change is made it will be going in some
    >> future release like 4.11; the problem still remains for the releases that
    >> are already out there. In case access to download.cloud.com is dropped
    >> at some point we will have to document the procedure such that people know
    >> how to make these stable releases work. That is where the procedure
    >> documented here https://shankerbalan.net/blog/
    >> seed-cloudstack-templates-offline/ should be officially adapted with a
    >> marketplace to pick up templates of choosing. Maybe we can have a pre-setup
    >> script to make choices instead of written wiki. Note that this will be
    >> anyway needed by the older releases.
    >>
    >>
    >>
    >>
    >> On 01/03/17, 4:24 PM, "Paul Angus" <pa...@shapeblue.com> wrote:
    >>
    >> >-1
    >> >
    >> >We are trying to get MORE people to use CloudStack.  And raising the
    >> barrier to entry does nothing to help that.
    >> >
    >> >
    >> >Kind regards,
    >> >
    >> >Paul Angus
    >> >
    >> >paul.angus@shapeblue.com
    >> >www.shapeblue.com
    >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
    >> >@shapeblue
    >> >
    >> >
    >> >
    >> >
    >> >-----Original Message-----
    >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@shapeblue.com]
    >> >Sent: 01 March 2017 09:39
    >> >To: dev@cloudstack.apache.org
    >> >Subject: Re: Modern template hosting
    >> >
    >> >+1
    >> >
    >> >After seeing this discussion running in circles several times, I think
    >> we should at least get started with the simplest option.
    >> >
    >> >
    >> >
    >> >
    >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:
    >> >
    >> >>The alternative foolproof way is to simply not provide the automatic
    >> >>download.
    >> >>
    >> >>Just document the mirror list URL where the templates can be found. I
    >> >>don't know why people reject this option.
    >> >>
    >> >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
    >> >>
    >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
    >> >><wi...@gmail.com>
    >> >>wrote:
    >> >>
    >> >>> OK. Thanks for the heads up.
    >> >>>
    >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <ch...@gmail.com>
    >> wrote:
    >> >>>
    >> >>> > Sounds workable. The downloader code in the SSVM won't follow
    >> >>> > redirects I think.
    >> >>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d9cd0444
    >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
    >> >>> > HttpTemplateDownloader.java#L93
    >> >>> > https://goo.gl/dSi0r5
    >> >>> >  Might need to add
    >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
    >> >>> >
    >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
    >> >>> > <ws...@cloudops.com>
    >> >>> > wrote:
    >> >>> >
    >> >>> > > We haven't opened a ticket yet because we don't have a strategy
    >> yet.
    >> >>> > >
    >> >>> > > What do you guys think of this:
    >> >>> > > - We setup a new github repo in the 'apache' org which consists
    >> >>> > > of a
    >> >>> > single
    >> >>> > > file with a list of active/supported mirrors.
    >> >>> > > - I write a small web server, distributed as a binary, which can
    >> >>> > > be
    >> >>> > hosted
    >> >>> > > by ASF Infra.  This web server will query the current list of
    >> >>> > > mirrors
    >> >>> and
    >> >>> > > will select one and then do a 302 redirect to that mirror.
    >> >>> > >
    >> >>> > > The act of 'choosing' a mirror could be done in a number of ways.
    >> >>> > > - If we want to define an order, then it could just try from the
    >> >>> > > top of
    >> >>> > the
    >> >>> > > list and work its way down.  It would curl the target to make
    >> >>> > > sure it
    >> >>> > gets
    >> >>> > > a 200 and if it does, it would do a 302 redirect.
    >> >>> > > - Or, if we want to distribute the load across the mirrors, we
    >> >>> > > could
    >> >>> pick
    >> >>> > > from the list randomly.  Again, doing a curl to verify the mirror
    >> >>> > > is up
    >> >>> > and
    >> >>> > > then doing a redirect.
    >> >>> > > - If we want to get fancy, we could do a reverse IP lookup and
    >> >>> > > try to
    >> >>> > match
    >> >>> > > the requester with their closest geographical mirror.
    >> >>> > >
    >> >>> > > Thoughts?
    >> >>> > >
    >> >>> > > *Will STEVENS*
    >> >>> > > Lead Developer
    >> >>> > >
    >> >>> > > <https://goo.gl/NYZ8KK>
    >> >>> > >
    >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
    >> >>> chiradeepv@gmail.com
    >> >>> > >
    >> >>> > > wrote:
    >> >>> > >
    >> >>> > > > What steps are needed to set up a mirror? What does Infra need
    >> to do?
    >> >>> > Has
    >> >>> > > > anybody filed a ticket with Infra?
    >> >>> > > >
    >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
    >> >>> > > > raja.pullela@accelerite.com>
    >> >>> > > > wrote:
    >> >>> > > >
    >> >>> > > > > Hi will,
    >> >>> > > > >
    >> >>> > > > > I believe, we didn’t get to close ‘getting a mirror on Apache’
    >> >>> > because
    >> >>> > > we
    >> >>> > > > > needed someone on the Apache Infra side to close this. BTW,
    >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
    >> >>> > > > > all/most of
    >> >>> > the
    >> >>> > > > > content.  Once we can close on the Apache mirror for hosting
    >> >>> > > > > the
    >> >>> > > > content, I
    >> >>> > > > > can help assist getting the content there.
    >> >>> > > > >
    >> >>> > > > > For now, we have replicated the download.cloud.com content to
    >> >>> > > > > ‘ s3.download.accelerite.com’.
    >> >>> > > > > Also, we are working on a set of steps/procedure to help with
    >> >>> > > > > this change.  I will update everyone in about a week’s time
    >> >>> > > > > on the
    >> >>> > details.
    >> >>> > > > >
    >> >>> > > > > Best,
    >> >>> > > > > Raja Pullela
    >> >>> > > > > Engineering Team,
    >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA, 95054
    >> >>> > > > >
    >> >>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of
    >> >>> > > > > Will Stevens" <williamstevens@gmail.com on behalf of
    >> >>> > wstevens@cloudops.com>
    >> >>> > > > > wrote:
    >> >>> > > > >
    >> >>> > > > > unfortunately the template mirror conversation got caught up
    >> >>> > > > > in
    >> >>> > details
    >> >>> > > > and
    >> >>> > > > > nobody took the lead on implementing a solution.
    >> >>> > > > >
    >> >>> > > > > citrix has been pinging me every couple months to say 'dude,
    >> >>> > > > > we
    >> >>> need
    >> >>> > to
    >> >>> > > > > remove the dependency on download.citrix.com', but i have not
    >> >>> > > > > had
    >> >>> > the
    >> >>> > > > > cycles to get in and solve the problem.  the shutdown of that
    >> >>> > > > > is
    >> >>> > > imminent
    >> >>> > > > > right now, so we need to solve it asap.
    >> >>> > > > >
    >> >>> > > > >
    >> >>> > > > >
    >> >>> > > > > *Will STEVENS*
    >> >>> > > > > Lead Developer
    >> >>> > > > >
    >> >>> > > > > <https://goo.gl/NYZ8KK>
    >> >>> > > > >
    >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
    >> >>> > paul.angus@shapeblue.com
    >> >>> > > >
    >> >>> > > > > wrote:
    >> >>> > > > >
    >> >>> > > > > > Hi Nathan,
    >> >>> > > > > >
    >> >>> > > > > > Ideally, if you put the template location in (or use a
    >> >>> > > > > > template
    >> >>> > > defined
    >> >>> > > > > > in)  test_data.py then the actual location can be
    >> >>> > > > > > overridden by
    >> >>> > > anyone
    >> >>> > > > > > testing.
    >> >>> > > > > >
    >> >>> > > > > > For Trillian, we've copied all of the templates that people
    >> >>> > > > > > have
    >> >>> > > define
    >> >>> > > > > to
    >> >>> > > > > > a local repo and then replace the URLs in test_data.py to
    >> >>> > > > > > reduce
    >> >>> > > > > bandwidth
    >> >>> > > > > > use and download times.
    >> >>> > > > > >
    >> >>> > > > > > Ie:
    >> >>> > > > > >
    >> >>> > > > > >             "bootableIso":
    >> >>> > > > > >                 {
    >> >>> > > > > >                     "displaytext": "Test Bootable ISO",
    >> >>> > > > > >                     "name": "testISO",
    >> >>> > > > > >                     "bootable": True,
    >> >>> > > > > >                     "ispublic": False,
    >> >>> > > > > >                     "url": "{{ marvin_images_location
    >> >>> > > > > > }}/TinyCore-current.iso",
    >> >>> > > > > >                     "ostype": 'Other Linux (64-bit)',
    >> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
    >> >>> > > > > >         },
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > > I thought that we had come up with a solution for
    >> >>> > download.cloud.com
    >> >>> > > ,
    >> >>> > > > by
    >> >>> > > > > > having a mirrorlist hosted in Community Apache 'space' with
    >> >>> anyone
    >> >>> > > able
    >> >>> > > > > to
    >> >>> > > > > > out themselves forward as a mirror.
    >> >>> > > > > > But I must admit I lost track of whether anyone made the
    >> >>> requisite
    >> >>> > > > > changes
    >> >>> > > > > > in code....
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > > Kind regards,
    >> >>> > > > > >
    >> >>> > > > > > Paul Angus
    >> >>> > > > > >
    >> >>> > > > > > paul.angus@shapeblue.com
    >> >>> > > > > > www.shapeblue.com
    >> >>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
    >> >>> > > > > > @shapeblue
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > > -----Original Message-----
    >> >>> > > > > > From: williamstevens@gmail.com
    >> >>> > > > > > [mailto:williamstevens@gmail.com]
    >> >>> > On
    >> >>> > > > > > Behalf Of Will Stevens
    >> >>> > > > > > Sent: 24 February 2017 16:30
    >> >>> > > > > > To: dev@cloudstack.apache.org
    >> >>> > > > > > Subject: Re: Modern template hosting
    >> >>> > > > > >
    >> >>> > > > > > this is a hard questions.  in general, we should be setting
    >> >>> > > > > > up a
    >> >>> > > mirror
    >> >>> > > > > on
    >> >>> > > > > > some cloudstack/apache domain and then mirror to other
    >> >>> > > > > > provided
    >> >>> > > > > templates.
    >> >>> > > > > >
    >> >>> > > > > > we MUST come up with a solution to deprecate
    >> >>> > > > > > 'download.cloud.com
    >> >>> ',
    >> >>> > > > that
    >> >>> > > > > > is going to be going away any day now.
    >> >>> > > > > >
    >> >>> > > > > > i don't know the right way to solve this to be honest, but
    >> >>> > > > > > if you
    >> >>> > > have
    >> >>> > > > > > ideas, i am willing to help.
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > > *Will STEVENS*
    >> >>> > > > > > Lead Developer
    >> >>> > > > > >
    >> >>> > > > > > <https://goo.gl/NYZ8KK>
    >> >>> > > > > >
    >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
    >> >>> njohnson@ena.com
    >> >>> > >
    >> >>> > > > > wrote:
    >> >>> > > > > >
    >> >>> > > > > > > So not to re-open a can of worms, but I’m in a situation
    >> >>> > > > > > > where
    >> >>> I
    >> >>> > > need
    >> >>> > > > > > > to come up with a Marvin component test that depends on a
    >> >>> > template
    >> >>> > > > > > > based on a kernel that’s relatively new, i.e., newer than
    >> >>> Centos
    >> >>> > > 5.3
    >> >>> > > > /
    >> >>> > > > > > Ubuntu 10.04 .
    >> >>> > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4
    >> >>> > > > > > > for
    >> >>> KVM),
    >> >>> > > but
    >> >>> > > > > > > from looking at the thread "Migrating CloudStack content
    >> >>> > > > > > > from download.cloud.com” it looks like there is
    >> >>> > > > > > > resistance to using
    >> >>> > > this
    >> >>> > > > at
    >> >>> > > > > > > least for hosting system vm templates over concerns of
    >> >>> > neutrality.
    >> >>> > > > > > > Would this be suitable for a component test?  If not,
    >> >>> > > > > > > what is a
    >> >>> > > > > “blessed”
    >> >>> > > > > > > template location?
    >> >>> > > > > > >
    >> >>> > > > > > > Thanks in advance!
    >> >>> > > > > > >
    >> >>> > > > > > > Nathan Johnson
    >> >>> > > > > > > R&D Engineer
    >> >>> > > > > > >
    >> >>> > > > > > >
    >> >>> > > > > > >
    >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville, TN 37211
    >> >>> > > > > > > General Office: 615-312-6000
    >> >>> > > > > > >
    >> >>> > > > > > > website | blog | support
    >> >>> > > > > > >
    >> >>> > > > > > >
    >> >>> > > > > > >
    >> >>> > > > > > >
    >> >>> > > > > > >
    >> >>> > > > > > >
    >> >>> > > > > > >
    >> >>> > > > > >
    >> >>> > > > >
    >> >>> > > > >
    >> >>> > > > >
    >> >>> > > > >
    >> >>> > > > >
    >> >>> > > > >
    >> >>> > > > > DISCLAIMER
    >> >>> > > > > ==========
    >> >>> > > > > This e-mail may contain privileged and confidential
    >> >>> > > > > information
    >> >>> which
    >> >>> > > is
    >> >>> > > > > the property of Accelerite, a Persistent Systems business. It
    >> >>> > > > > is
    >> >>> > > intended
    >> >>> > > > > only for the use of the individual or entity to which it is
    >> >>> > addressed.
    >> >>> > > If
    >> >>> > > > > you are not the intended recipient, you are not authorized to
    >> >>> > > > > read,
    >> >>> > > > retain,
    >> >>> > > > > copy, print, distribute or use this message. If you have
    >> >>> > > > > received
    >> >>> > this
    >> >>> > > > > communication in error, please notify the sender and delete
    >> >>> > > > > all
    >> >>> > copies
    >> >>> > > of
    >> >>> > > > > this message. Accelerite, a Persistent Systems business does
    >> >>> > > > > not
    >> >>> > accept
    >> >>> > > > any
    >> >>> > > > > liability for virus infected mails.
    >> >>> > > > >
    >> >>> > > >
    >> >>> > >
    >> >>> >
    >> >>>
    >> >
    >> >abhinandan.prateek@shapeblue.com
    >> >www.shapeblue.com
    >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
    >> >
    >> >
    >> >
    >>
    >> abhinandan.prateek@shapeblue.com
    >> www.shapeblue.com
    >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
    >> @shapeblue
    >>
    >>
    >>
    >>
    >
    


daan.hoogland@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue

Re: Modern template hosting

Posted by Will Stevens <sw...@apache.org>.

Hey All,
Please review this repo and tell me what you think:
https://github.com/swill/mirror302

Let me know if you have questions or you would like me to make changes
before I bring this topic up with ASF Infra and open a ticket to get this
implemented.

Cheers,

*Will Stevens*



On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <ws...@cloudops.com> wrote:

> I am building a short term solution right now.  I hopefully will have the
> building blocks in place by the weekend so we can start working with Infra
> to get it in place.
>
> We will have a web server which we can point something like
> downloads.cloudstack.org (or whatever url) at.  Then we will have an ASF
> repo which tracks a mirror list and exposes a static site to be able to
> query the mirror list.
>
> The web server will receive a request and will do a 302 redirect to the
> appropriate resource on one of the mirrors.
>
> This gets us started.  Legacy environments will have to do a DB change to
> change from 'download.cloud.com' to 'downloads.cloudstack.org' (or
> whatever), but otherwise it should be pretty simple.
>
> We can improve how we deliver templates going forward, but this obviously
> requires some discussion still.  I will try to buy us some time with an
> implementation which solves our problems today...
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> abhinandan.prateek@shapeblue.com> wrote:
>
>>
>> Initial seeding is a manual step and that is not going to change in near
>> future. A handy list of official places from where these templates can be
>> downloaded won’t harm this part of the installation. Just allow anyone
>> credible who follows democratic process to publish their templates with
>> apache’s blessings.
>>
>>
>> Coming to urls hard coded in db, the popular suggestion is to point them
>> to mirrors. This looks good but this is going to take some effort that has
>> not materialised yet. Once this change is made it will be going in some
>> future release like 4.11; the problem still remains for the releases that
>> are already out there. In case access to download.cloud.com is dropped
>> at some point we will have to document the procedure such that people know
>> how to make these stable releases work. That is where the procedure
>> documented here https://shankerbalan.net/blog/
>> seed-cloudstack-templates-offline/ should be officially adapted with a
>> marketplace to pick up templates of choosing. Maybe we can have a pre-setup
>> script to make choices instead of written wiki. Note that this will be
>> anyway needed by the older releases.
>>
>>
>>
>>
>> On 01/03/17, 4:24 PM, "Paul Angus" <pa...@shapeblue.com> wrote:
>>
>> >-1
>> >
>> >We are trying to get MORE people to use CloudStack.  And raising the
>> barrier to entry does nothing to help that.
>> >
>> >
>> >Kind regards,
>> >
>> >Paul Angus
>> >
>> >paul.angus@shapeblue.com
>> >www.shapeblue.com
>> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> >@shapeblue
>> >
>> >
>> >
>> >
>> >-----Original Message-----
>> >From: Abhinandan Prateek [mailto:abhinandan.prateek@shapeblue.com]
>> >Sent: 01 March 2017 09:39
>> >To: dev@cloudstack.apache.org
>> >Subject: Re: Modern template hosting
>> >
>> >+1
>> >
>> >After seeing this discussion running in circles several times, I think
>> we should at least get started with the simplest option.
>> >
>> >
>> >
>> >
>> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:
>> >
>> >>The alternative foolproof way is to simply not provide the automatic
>> >>download.
>> >>
>> >>Just document the mirror list URL where the templates can be found. I
>> >>don't know why people reject this option.
>> >>
>> >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
>> >>
>> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
>> >><wi...@gmail.com>
>> >>wrote:
>> >>
>> >>> OK. Thanks for the heads up.
>> >>>
>> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <ch...@gmail.com>
>> wrote:
>> >>>
>> >>> > Sounds workable. The downloader code in the SSVM won't follow
>> >>> > redirects I think.
>> >>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d9cd0444
>> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
>> >>> > HttpTemplateDownloader.java#L93
>> >>> > https://goo.gl/dSi0r5
>> >>> >  Might need to add
>> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
>> >>> >
>> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
>> >>> > <ws...@cloudops.com>
>> >>> > wrote:
>> >>> >
>> >>> > > We haven't opened a ticket yet because we don't have a strategy
>> yet.
>> >>> > >
>> >>> > > What do you guys think of this:
>> >>> > > - We setup a new github repo in the 'apache' org which consists
>> >>> > > of a
>> >>> > single
>> >>> > > file with a list of active/supported mirrors.
>> >>> > > - I write a small web server, distributed as a binary, which can
>> >>> > > be
>> >>> > hosted
>> >>> > > by ASF Infra.  This web server will query the current list of
>> >>> > > mirrors
>> >>> and
>> >>> > > will select one and then do a 302 redirect to that mirror.
>> >>> > >
>> >>> > > The act of 'choosing' a mirror could be done in a number of ways.
>> >>> > > - If we want to define an order, then it could just try from the
>> >>> > > top of
>> >>> > the
>> >>> > > list and work its way down.  It would curl the target to make
>> >>> > > sure it
>> >>> > gets
>> >>> > > a 200 and if it does, it would do a 302 redirect.
>> >>> > > - Or, if we want to distribute the load across the mirrors, we
>> >>> > > could
>> >>> pick
>> >>> > > from the list randomly.  Again, doing a curl to verify the mirror
>> >>> > > is up
>> >>> > and
>> >>> > > then doing a redirect.
>> >>> > > - If we want to get fancy, we could do a reverse IP lookup and
>> >>> > > try to
>> >>> > match
>> >>> > > the requester with their closest geographical mirror.
>> >>> > >
>> >>> > > Thoughts?
>> >>> > >
>> >>> > > *Will STEVENS*
>> >>> > > Lead Developer
>> >>> > >
>> >>> > > <https://goo.gl/NYZ8KK>
>> >>> > >
>> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
>> >>> chiradeepv@gmail.com
>> >>> > >
>> >>> > > wrote:
>> >>> > >
>> >>> > > > What steps are needed to set up a mirror? What does Infra need
>> to do?
>> >>> > Has
>> >>> > > > anybody filed a ticket with Infra?
>> >>> > > >
>> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
>> >>> > > > raja.pullela@accelerite.com>
>> >>> > > > wrote:
>> >>> > > >
>> >>> > > > > Hi will,
>> >>> > > > >
>> >>> > > > > I believe, we didn’t get to close ‘getting a mirror on Apache’
>> >>> > because
>> >>> > > we
>> >>> > > > > needed someone on the Apache Infra side to close this. BTW,
>> >>> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
>> >>> > > > > all/most of
>> >>> > the
>> >>> > > > > content.  Once we can close on the Apache mirror for hosting
>> >>> > > > > the
>> >>> > > > content, I
>> >>> > > > > can help assist getting the content there.
>> >>> > > > >
>> >>> > > > > For now, we have replicated the download.cloud.com content to
>> >>> > > > > ‘ s3.download.accelerite.com’.
>> >>> > > > > Also, we are working on a set of steps/procedure to help with
>> >>> > > > > this change.  I will update everyone in about a week’s time
>> >>> > > > > on the
>> >>> > details.
>> >>> > > > >
>> >>> > > > > Best,
>> >>> > > > > Raja Pullela
>> >>> > > > > Engineering Team,
>> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA, 95054
>> >>> > > > >
>> >>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of
>> >>> > > > > Will Stevens" <williamstevens@gmail.com on behalf of
>> >>> > wstevens@cloudops.com>
>> >>> > > > > wrote:
>> >>> > > > >
>> >>> > > > > unfortunately the template mirror conversation got caught up
>> >>> > > > > in
>> >>> > details
>> >>> > > > and
>> >>> > > > > nobody took the lead on implementing a solution.
>> >>> > > > >
>> >>> > > > > citrix has been pinging me every couple months to say 'dude,
>> >>> > > > > we
>> >>> need
>> >>> > to
>> >>> > > > > remove the dependency on download.citrix.com', but i have not
>> >>> > > > > had
>> >>> > the
>> >>> > > > > cycles to get in and solve the problem.  the shutdown of that
>> >>> > > > > is
>> >>> > > imminent
>> >>> > > > > right now, so we need to solve it asap.
>> >>> > > > >
>> >>> > > > >
>> >>> > > > >
>> >>> > > > > *Will STEVENS*
>> >>> > > > > Lead Developer
>> >>> > > > >
>> >>> > > > > <https://goo.gl/NYZ8KK>
>> >>> > > > >
>> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
>> >>> > paul.angus@shapeblue.com
>> >>> > > >
>> >>> > > > > wrote:
>> >>> > > > >
>> >>> > > > > > Hi Nathan,
>> >>> > > > > >
>> >>> > > > > > Ideally, if you put the template location in (or use a
>> >>> > > > > > template
>> >>> > > defined
>> >>> > > > > > in)  test_data.py then the actual location can be
>> >>> > > > > > overridden by
>> >>> > > anyone
>> >>> > > > > > testing.
>> >>> > > > > >
>> >>> > > > > > For Trillian, we've copied all of the templates that people
>> >>> > > > > > have
>> >>> > > define
>> >>> > > > > to
>> >>> > > > > > a local repo and then replace the URLs in test_data.py to
>> >>> > > > > > reduce
>> >>> > > > > bandwidth
>> >>> > > > > > use and download times.
>> >>> > > > > >
>> >>> > > > > > Ie:
>> >>> > > > > >
>> >>> > > > > >             "bootableIso":
>> >>> > > > > >                 {
>> >>> > > > > >                     "displaytext": "Test Bootable ISO",
>> >>> > > > > >                     "name": "testISO",
>> >>> > > > > >                     "bootable": True,
>> >>> > > > > >                     "ispublic": False,
>> >>> > > > > >                     "url": "{{ marvin_images_location
>> >>> > > > > > }}/TinyCore-current.iso",
>> >>> > > > > >                     "ostype": 'Other Linux (64-bit)',
>> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
>> >>> > > > > >         },
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > > I thought that we had come up with a solution for
>> >>> > download.cloud.com
>> >>> > > ,
>> >>> > > > by
>> >>> > > > > > having a mirrorlist hosted in Community Apache 'space' with
>> >>> anyone
>> >>> > > able
>> >>> > > > > to
>> >>> > > > > > out themselves forward as a mirror.
>> >>> > > > > > But I must admit I lost track of whether anyone made the
>> >>> requisite
>> >>> > > > > changes
>> >>> > > > > > in code....
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > > Kind regards,
>> >>> > > > > >
>> >>> > > > > > Paul Angus
>> >>> > > > > >
>> >>> > > > > > paul.angus@shapeblue.com
>> >>> > > > > > www.shapeblue.com
>> >>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> >>> > > > > > @shapeblue
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > > -----Original Message-----
>> >>> > > > > > From: williamstevens@gmail.com
>> >>> > > > > > [mailto:williamstevens@gmail.com]
>> >>> > On
>> >>> > > > > > Behalf Of Will Stevens
>> >>> > > > > > Sent: 24 February 2017 16:30
>> >>> > > > > > To: dev@cloudstack.apache.org
>> >>> > > > > > Subject: Re: Modern template hosting
>> >>> > > > > >
>> >>> > > > > > this is a hard questions.  in general, we should be setting
>> >>> > > > > > up a
>> >>> > > mirror
>> >>> > > > > on
>> >>> > > > > > some cloudstack/apache domain and then mirror to other
>> >>> > > > > > provided
>> >>> > > > > templates.
>> >>> > > > > >
>> >>> > > > > > we MUST come up with a solution to deprecate
>> >>> > > > > > 'download.cloud.com
>> >>> ',
>> >>> > > > that
>> >>> > > > > > is going to be going away any day now.
>> >>> > > > > >
>> >>> > > > > > i don't know the right way to solve this to be honest, but
>> >>> > > > > > if you
>> >>> > > have
>> >>> > > > > > ideas, i am willing to help.
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > > *Will STEVENS*
>> >>> > > > > > Lead Developer
>> >>> > > > > >
>> >>> > > > > > <https://goo.gl/NYZ8KK>
>> >>> > > > > >
>> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
>> >>> njohnson@ena.com
>> >>> > >
>> >>> > > > > wrote:
>> >>> > > > > >
>> >>> > > > > > > So not to re-open a can of worms, but I’m in a situation
>> >>> > > > > > > where
>> >>> I
>> >>> > > need
>> >>> > > > > > > to come up with a Marvin component test that depends on a
>> >>> > template
>> >>> > > > > > > based on a kernel that’s relatively new, i.e., newer than
>> >>> Centos
>> >>> > > 5.3
>> >>> > > > /
>> >>> > > > > > Ubuntu 10.04 .
>> >>> > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4
>> >>> > > > > > > for
>> >>> KVM),
>> >>> > > but
>> >>> > > > > > > from looking at the thread "Migrating CloudStack content
>> >>> > > > > > > from download.cloud.com” it looks like there is
>> >>> > > > > > > resistance to using
>> >>> > > this
>> >>> > > > at
>> >>> > > > > > > least for hosting system vm templates over concerns of
>> >>> > neutrality.
>> >>> > > > > > > Would this be suitable for a component test?  If not,
>> >>> > > > > > > what is a
>> >>> > > > > “blessed”
>> >>> > > > > > > template location?
>> >>> > > > > > >
>> >>> > > > > > > Thanks in advance!
>> >>> > > > > > >
>> >>> > > > > > > Nathan Johnson
>> >>> > > > > > > R&D Engineer
>> >>> > > > > > >
>> >>> > > > > > >
>> >>> > > > > > >
>> >>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville, TN 37211
>> >>> > > > > > > General Office: 615-312-6000
>> >>> > > > > > >
>> >>> > > > > > > website | blog | support
>> >>> > > > > > >
>> >>> > > > > > >
>> >>> > > > > > >
>> >>> > > > > > >
>> >>> > > > > > >
>> >>> > > > > > >
>> >>> > > > > > >
>> >>> > > > > >
>> >>> > > > >
>> >>> > > > >
>> >>> > > > >
>> >>> > > > >
>> >>> > > > >
>> >>> > > > >
>> >>> > > > > DISCLAIMER
>> >>> > > > > ==========
>> >>> > > > > This e-mail may contain privileged and confidential
>> >>> > > > > information
>> >>> which
>> >>> > > is
>> >>> > > > > the property of Accelerite, a Persistent Systems business. It
>> >>> > > > > is
>> >>> > > intended
>> >>> > > > > only for the use of the individual or entity to which it is
>> >>> > addressed.
>> >>> > > If
>> >>> > > > > you are not the intended recipient, you are not authorized to
>> >>> > > > > read,
>> >>> > > > retain,
>> >>> > > > > copy, print, distribute or use this message. If you have
>> >>> > > > > received
>> >>> > this
>> >>> > > > > communication in error, please notify the sender and delete
>> >>> > > > > all
>> >>> > copies
>> >>> > > of
>> >>> > > > > this message. Accelerite, a Persistent Systems business does
>> >>> > > > > not
>> >>> > accept
>> >>> > > > any
>> >>> > > > > liability for virus infected mails.
>> >>> > > > >
>> >>> > > >
>> >>> > >
>> >>> >
>> >>>
>> >
>> >abhinandan.prateek@shapeblue.com
>> >www.shapeblue.com
>> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>> >
>> >
>> >
>>
>> abhinandan.prateek@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> @shapeblue
>>
>>
>>
>>
>

Re: Modern template hosting

Posted by Will Stevens <ws...@cloudops.com>.

I am building a short term solution right now.  I hopefully will have the
building blocks in place by the weekend so we can start working with Infra
to get it in place.

We will have a web server which we can point something like
downloads.cloudstack.org (or whatever url) at.  Then we will have an ASF
repo which tracks a mirror list and exposes a static site to be able to
query the mirror list.

The web server will receive a request and will do a 302 redirect to the
appropriate resource on one of the mirrors.

This gets us started.  Legacy environments will have to do a DB change to
change from 'download.cloud.com' to 'downloads.cloudstack.org' (or
whatever), but otherwise it should be pretty simple.

We can improve how we deliver templates going forward, but this obviously
requires some discussion still.  I will try to buy us some time with an
implementation which solves our problems today...

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
abhinandan.prateek@shapeblue.com> wrote:

>
> Initial seeding is a manual step and that is not going to change in near
> future. A handy list of official places from where these templates can be
> downloaded won’t harm this part of the installation. Just allow anyone
> credible who follows democratic process to publish their templates with
> apache’s blessings.
>
>
> Coming to urls hard coded in db, the popular suggestion is to point them
> to mirrors. This looks good but this is going to take some effort that has
> not materialised yet. Once this change is made it will be going in some
> future release like 4.11; the problem still remains for the releases that
> are already out there. In case access to download.cloud.com is dropped at
> some point we will have to document the procedure such that people know how
> to make these stable releases work. That is where the procedure documented
> here https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
> should be officially adapted with a marketplace to pick up templates of
> choosing. Maybe we can have a pre-setup script to make choices instead of
> written wiki. Note that this will be anyway needed by the older releases.
>
>
>
>
> On 01/03/17, 4:24 PM, "Paul Angus" <pa...@shapeblue.com> wrote:
>
> >-1
> >
> >We are trying to get MORE people to use CloudStack.  And raising the
> barrier to entry does nothing to help that.
> >
> >
> >Kind regards,
> >
> >Paul Angus
> >
> >paul.angus@shapeblue.com
> >www.shapeblue.com
> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >@shapeblue
> >
> >
> >
> >
> >-----Original Message-----
> >From: Abhinandan Prateek [mailto:abhinandan.prateek@shapeblue.com]
> >Sent: 01 March 2017 09:39
> >To: dev@cloudstack.apache.org
> >Subject: Re: Modern template hosting
> >
> >+1
> >
> >After seeing this discussion running in circles several times, I think we
> should at least get started with the simplest option.
> >
> >
> >
> >
> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:
> >
> >>The alternative foolproof way is to simply not provide the automatic
> >>download.
> >>
> >>Just document the mirror list URL where the templates can be found. I
> >>don't know why people reject this option.
> >>
> >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
> >>
> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> >><wi...@gmail.com>
> >>wrote:
> >>
> >>> OK. Thanks for the heads up.
> >>>
> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <ch...@gmail.com>
> wrote:
> >>>
> >>> > Sounds workable. The downloader code in the SSVM won't follow
> >>> > redirects I think.
> >>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d9cd0444
> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
> >>> > HttpTemplateDownloader.java#L93
> >>> > https://goo.gl/dSi0r5
> >>> >  Might need to add
> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
> >>> >
> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> >>> > <ws...@cloudops.com>
> >>> > wrote:
> >>> >
> >>> > > We haven't opened a ticket yet because we don't have a strategy
> yet.
> >>> > >
> >>> > > What do you guys think of this:
> >>> > > - We setup a new github repo in the 'apache' org which consists
> >>> > > of a
> >>> > single
> >>> > > file with a list of active/supported mirrors.
> >>> > > - I write a small web server, distributed as a binary, which can
> >>> > > be
> >>> > hosted
> >>> > > by ASF Infra.  This web server will query the current list of
> >>> > > mirrors
> >>> and
> >>> > > will select one and then do a 302 redirect to that mirror.
> >>> > >
> >>> > > The act of 'choosing' a mirror could be done in a number of ways.
> >>> > > - If we want to define an order, then it could just try from the
> >>> > > top of
> >>> > the
> >>> > > list and work its way down.  It would curl the target to make
> >>> > > sure it
> >>> > gets
> >>> > > a 200 and if it does, it would do a 302 redirect.
> >>> > > - Or, if we want to distribute the load across the mirrors, we
> >>> > > could
> >>> pick
> >>> > > from the list randomly.  Again, doing a curl to verify the mirror
> >>> > > is up
> >>> > and
> >>> > > then doing a redirect.
> >>> > > - If we want to get fancy, we could do a reverse IP lookup and
> >>> > > try to
> >>> > match
> >>> > > the requester with their closest geographical mirror.
> >>> > >
> >>> > > Thoughts?
> >>> > >
> >>> > > *Will STEVENS*
> >>> > > Lead Developer
> >>> > >
> >>> > > <https://goo.gl/NYZ8KK>
> >>> > >
> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> >>> chiradeepv@gmail.com
> >>> > >
> >>> > > wrote:
> >>> > >
> >>> > > > What steps are needed to set up a mirror? What does Infra need
> to do?
> >>> > Has
> >>> > > > anybody filed a ticket with Infra?
> >>> > > >
> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> >>> > > > raja.pullela@accelerite.com>
> >>> > > > wrote:
> >>> > > >
> >>> > > > > Hi will,
> >>> > > > >
> >>> > > > > I believe, we didn’t get to close ‘getting a mirror on Apache’
> >>> > because
> >>> > > we
> >>> > > > > needed someone on the Apache Infra side to close this. BTW,
> >>> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
> >>> > > > > all/most of
> >>> > the
> >>> > > > > content.  Once we can close on the Apache mirror for hosting
> >>> > > > > the
> >>> > > > content, I
> >>> > > > > can help assist getting the content there.
> >>> > > > >
> >>> > > > > For now, we have replicated the download.cloud.com content to
> >>> > > > > ‘ s3.download.accelerite.com’.
> >>> > > > > Also, we are working on a set of steps/procedure to help with
> >>> > > > > this change.  I will update everyone in about a week’s time
> >>> > > > > on the
> >>> > details.
> >>> > > > >
> >>> > > > > Best,
> >>> > > > > Raja Pullela
> >>> > > > > Engineering Team,
> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA, 95054
> >>> > > > >
> >>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of
> >>> > > > > Will Stevens" <williamstevens@gmail.com on behalf of
> >>> > wstevens@cloudops.com>
> >>> > > > > wrote:
> >>> > > > >
> >>> > > > > unfortunately the template mirror conversation got caught up
> >>> > > > > in
> >>> > details
> >>> > > > and
> >>> > > > > nobody took the lead on implementing a solution.
> >>> > > > >
> >>> > > > > citrix has been pinging me every couple months to say 'dude,
> >>> > > > > we
> >>> need
> >>> > to
> >>> > > > > remove the dependency on download.citrix.com', but i have not
> >>> > > > > had
> >>> > the
> >>> > > > > cycles to get in and solve the problem.  the shutdown of that
> >>> > > > > is
> >>> > > imminent
> >>> > > > > right now, so we need to solve it asap.
> >>> > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > > *Will STEVENS*
> >>> > > > > Lead Developer
> >>> > > > >
> >>> > > > > <https://goo.gl/NYZ8KK>
> >>> > > > >
> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> >>> > paul.angus@shapeblue.com
> >>> > > >
> >>> > > > > wrote:
> >>> > > > >
> >>> > > > > > Hi Nathan,
> >>> > > > > >
> >>> > > > > > Ideally, if you put the template location in (or use a
> >>> > > > > > template
> >>> > > defined
> >>> > > > > > in)  test_data.py then the actual location can be
> >>> > > > > > overridden by
> >>> > > anyone
> >>> > > > > > testing.
> >>> > > > > >
> >>> > > > > > For Trillian, we've copied all of the templates that people
> >>> > > > > > have
> >>> > > define
> >>> > > > > to
> >>> > > > > > a local repo and then replace the URLs in test_data.py to
> >>> > > > > > reduce
> >>> > > > > bandwidth
> >>> > > > > > use and download times.
> >>> > > > > >
> >>> > > > > > Ie:
> >>> > > > > >
> >>> > > > > >             "bootableIso":
> >>> > > > > >                 {
> >>> > > > > >                     "displaytext": "Test Bootable ISO",
> >>> > > > > >                     "name": "testISO",
> >>> > > > > >                     "bootable": True,
> >>> > > > > >                     "ispublic": False,
> >>> > > > > >                     "url": "{{ marvin_images_location
> >>> > > > > > }}/TinyCore-current.iso",
> >>> > > > > >                     "ostype": 'Other Linux (64-bit)',
> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
> >>> > > > > >         },
> >>> > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > I thought that we had come up with a solution for
> >>> > download.cloud.com
> >>> > > ,
> >>> > > > by
> >>> > > > > > having a mirrorlist hosted in Community Apache 'space' with
> >>> anyone
> >>> > > able
> >>> > > > > to
> >>> > > > > > out themselves forward as a mirror.
> >>> > > > > > But I must admit I lost track of whether anyone made the
> >>> requisite
> >>> > > > > changes
> >>> > > > > > in code....
> >>> > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > Kind regards,
> >>> > > > > >
> >>> > > > > > Paul Angus
> >>> > > > > >
> >>> > > > > > paul.angus@shapeblue.com
> >>> > > > > > www.shapeblue.com
> >>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >>> > > > > > @shapeblue
> >>> > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > -----Original Message-----
> >>> > > > > > From: williamstevens@gmail.com
> >>> > > > > > [mailto:williamstevens@gmail.com]
> >>> > On
> >>> > > > > > Behalf Of Will Stevens
> >>> > > > > > Sent: 24 February 2017 16:30
> >>> > > > > > To: dev@cloudstack.apache.org
> >>> > > > > > Subject: Re: Modern template hosting
> >>> > > > > >
> >>> > > > > > this is a hard questions.  in general, we should be setting
> >>> > > > > > up a
> >>> > > mirror
> >>> > > > > on
> >>> > > > > > some cloudstack/apache domain and then mirror to other
> >>> > > > > > provided
> >>> > > > > templates.
> >>> > > > > >
> >>> > > > > > we MUST come up with a solution to deprecate
> >>> > > > > > 'download.cloud.com
> >>> ',
> >>> > > > that
> >>> > > > > > is going to be going away any day now.
> >>> > > > > >
> >>> > > > > > i don't know the right way to solve this to be honest, but
> >>> > > > > > if you
> >>> > > have
> >>> > > > > > ideas, i am willing to help.
> >>> > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > *Will STEVENS*
> >>> > > > > > Lead Developer
> >>> > > > > >
> >>> > > > > > <https://goo.gl/NYZ8KK>
> >>> > > > > >
> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> >>> njohnson@ena.com
> >>> > >
> >>> > > > > wrote:
> >>> > > > > >
> >>> > > > > > > So not to re-open a can of worms, but I’m in a situation
> >>> > > > > > > where
> >>> I
> >>> > > need
> >>> > > > > > > to come up with a Marvin component test that depends on a
> >>> > template
> >>> > > > > > > based on a kernel that’s relatively new, i.e., newer than
> >>> Centos
> >>> > > 5.3
> >>> > > > /
> >>> > > > > > Ubuntu 10.04 .
> >>> > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4
> >>> > > > > > > for
> >>> KVM),
> >>> > > but
> >>> > > > > > > from looking at the thread "Migrating CloudStack content
> >>> > > > > > > from download.cloud.com” it looks like there is
> >>> > > > > > > resistance to using
> >>> > > this
> >>> > > > at
> >>> > > > > > > least for hosting system vm templates over concerns of
> >>> > neutrality.
> >>> > > > > > > Would this be suitable for a component test?  If not,
> >>> > > > > > > what is a
> >>> > > > > “blessed”
> >>> > > > > > > template location?
> >>> > > > > > >
> >>> > > > > > > Thanks in advance!
> >>> > > > > > >
> >>> > > > > > > Nathan Johnson
> >>> > > > > > > R&D Engineer
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville, TN 37211
> >>> > > > > > > General Office: 615-312-6000
> >>> > > > > > >
> >>> > > > > > > website | blog | support
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > > DISCLAIMER
> >>> > > > > ==========
> >>> > > > > This e-mail may contain privileged and confidential
> >>> > > > > information
> >>> which
> >>> > > is
> >>> > > > > the property of Accelerite, a Persistent Systems business. It
> >>> > > > > is
> >>> > > intended
> >>> > > > > only for the use of the individual or entity to which it is
> >>> > addressed.
> >>> > > If
> >>> > > > > you are not the intended recipient, you are not authorized to
> >>> > > > > read,
> >>> > > > retain,
> >>> > > > > copy, print, distribute or use this message. If you have
> >>> > > > > received
> >>> > this
> >>> > > > > communication in error, please notify the sender and delete
> >>> > > > > all
> >>> > copies
> >>> > > of
> >>> > > > > this message. Accelerite, a Persistent Systems business does
> >>> > > > > not
> >>> > accept
> >>> > > > any
> >>> > > > > liability for virus infected mails.
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >
> >abhinandan.prateek@shapeblue.com
> >www.shapeblue.com
> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
> >
> >
> >
>
> abhinandan.prateek@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>

Re: Modern template hosting

Posted by Abhinandan Prateek <ab...@shapeblue.com>.

Initial seeding is a manual step and that is not going to change in near future. A handy list of official places from where these templates can be downloaded won’t harm this part of the installation. Just allow anyone credible who follows democratic process to publish their templates with apache’s blessings.


Coming to urls hard coded in db, the popular suggestion is to point them to mirrors. This looks good but this is going to take some effort that has not materialised yet. Once this change is made it will be going in some future release like 4.11; the problem still remains for the releases that are already out there. In case access to download.cloud.com is dropped at some point we will have to document the procedure such that people know how to make these stable releases work. That is where the procedure documented here https://shankerbalan.net/blog/seed-cloudstack-templates-offline/ should be officially adapted with a marketplace to pick up templates of choosing. Maybe we can have a pre-setup script to make choices instead of written wiki. Note that this will be anyway needed by the older releases.




On 01/03/17, 4:24 PM, "Paul Angus" <pa...@shapeblue.com> wrote:

>-1
>
>We are trying to get MORE people to use CloudStack.  And raising the barrier to entry does nothing to help that.
>
>
>Kind regards,
>
>Paul Angus
>
>paul.angus@shapeblue.com 
>www.shapeblue.com
>53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>@shapeblue
>  
> 
>
>
>-----Original Message-----
>From: Abhinandan Prateek [mailto:abhinandan.prateek@shapeblue.com] 
>Sent: 01 March 2017 09:39
>To: dev@cloudstack.apache.org
>Subject: Re: Modern template hosting
>
>+1
>
>After seeing this discussion running in circles several times, I think we should at least get started with the simplest option.
>
>
>
>
>On 27/02/17, 11:54 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:
>
>>The alternative foolproof way is to simply not provide the automatic 
>>download.
>>
>>Just document the mirror list URL where the templates can be found. I 
>>don't know why people reject this option.
>>
>>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
>>
>>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens 
>><wi...@gmail.com>
>>wrote:
>>
>>> OK. Thanks for the heads up.
>>>
>>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:
>>>
>>> > Sounds workable. The downloader code in the SSVM won't follow 
>>> > redirects I think.
>>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d9cd0444
>>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
>>> > HttpTemplateDownloader.java#L93
>>> > https://goo.gl/dSi0r5
>>> >  Might need to add
>>> > client.setRedirectStrategy(new LaxRedirectStrategy());
>>> >
>>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens 
>>> > <ws...@cloudops.com>
>>> > wrote:
>>> >
>>> > > We haven't opened a ticket yet because we don't have a strategy yet.
>>> > >
>>> > > What do you guys think of this:
>>> > > - We setup a new github repo in the 'apache' org which consists 
>>> > > of a
>>> > single
>>> > > file with a list of active/supported mirrors.
>>> > > - I write a small web server, distributed as a binary, which can 
>>> > > be
>>> > hosted
>>> > > by ASF Infra.  This web server will query the current list of 
>>> > > mirrors
>>> and
>>> > > will select one and then do a 302 redirect to that mirror.
>>> > >
>>> > > The act of 'choosing' a mirror could be done in a number of ways.
>>> > > - If we want to define an order, then it could just try from the 
>>> > > top of
>>> > the
>>> > > list and work its way down.  It would curl the target to make 
>>> > > sure it
>>> > gets
>>> > > a 200 and if it does, it would do a 302 redirect.
>>> > > - Or, if we want to distribute the load across the mirrors, we 
>>> > > could
>>> pick
>>> > > from the list randomly.  Again, doing a curl to verify the mirror 
>>> > > is up
>>> > and
>>> > > then doing a redirect.
>>> > > - If we want to get fancy, we could do a reverse IP lookup and 
>>> > > try to
>>> > match
>>> > > the requester with their closest geographical mirror.
>>> > >
>>> > > Thoughts?
>>> > >
>>> > > *Will STEVENS*
>>> > > Lead Developer
>>> > >
>>> > > <https://goo.gl/NYZ8KK>
>>> > >
>>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
>>> chiradeepv@gmail.com
>>> > >
>>> > > wrote:
>>> > >
>>> > > > What steps are needed to set up a mirror? What does Infra need to do?
>>> > Has
>>> > > > anybody filed a ticket with Infra?
>>> > > >
>>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela < 
>>> > > > raja.pullela@accelerite.com>
>>> > > > wrote:
>>> > > >
>>> > > > > Hi will,
>>> > > > >
>>> > > > > I believe, we didn’t get to close ‘getting a mirror on Apache’
>>> > because
>>> > > we
>>> > > > > needed someone on the Apache Infra side to close this. BTW, 
>>> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has 
>>> > > > > all/most of
>>> > the
>>> > > > > content.  Once we can close on the Apache mirror for hosting 
>>> > > > > the
>>> > > > content, I
>>> > > > > can help assist getting the content there.
>>> > > > >
>>> > > > > For now, we have replicated the download.cloud.com content to 
>>> > > > > ‘ s3.download.accelerite.com’.
>>> > > > > Also, we are working on a set of steps/procedure to help with 
>>> > > > > this change.  I will update everyone in about a week’s time 
>>> > > > > on the
>>> > details.
>>> > > > >
>>> > > > > Best,
>>> > > > > Raja Pullela
>>> > > > > Engineering Team,
>>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA, 95054
>>> > > > >
>>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of 
>>> > > > > Will Stevens" <williamstevens@gmail.com on behalf of
>>> > wstevens@cloudops.com>
>>> > > > > wrote:
>>> > > > >
>>> > > > > unfortunately the template mirror conversation got caught up 
>>> > > > > in
>>> > details
>>> > > > and
>>> > > > > nobody took the lead on implementing a solution.
>>> > > > >
>>> > > > > citrix has been pinging me every couple months to say 'dude, 
>>> > > > > we
>>> need
>>> > to
>>> > > > > remove the dependency on download.citrix.com', but i have not 
>>> > > > > had
>>> > the
>>> > > > > cycles to get in and solve the problem.  the shutdown of that 
>>> > > > > is
>>> > > imminent
>>> > > > > right now, so we need to solve it asap.
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > *Will STEVENS*
>>> > > > > Lead Developer
>>> > > > >
>>> > > > > <https://goo.gl/NYZ8KK>
>>> > > > >
>>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
>>> > paul.angus@shapeblue.com
>>> > > >
>>> > > > > wrote:
>>> > > > >
>>> > > > > > Hi Nathan,
>>> > > > > >
>>> > > > > > Ideally, if you put the template location in (or use a 
>>> > > > > > template
>>> > > defined
>>> > > > > > in)  test_data.py then the actual location can be 
>>> > > > > > overridden by
>>> > > anyone
>>> > > > > > testing.
>>> > > > > >
>>> > > > > > For Trillian, we've copied all of the templates that people 
>>> > > > > > have
>>> > > define
>>> > > > > to
>>> > > > > > a local repo and then replace the URLs in test_data.py to 
>>> > > > > > reduce
>>> > > > > bandwidth
>>> > > > > > use and download times.
>>> > > > > >
>>> > > > > > Ie:
>>> > > > > >
>>> > > > > >             "bootableIso":
>>> > > > > >                 {
>>> > > > > >                     "displaytext": "Test Bootable ISO",
>>> > > > > >                     "name": "testISO",
>>> > > > > >                     "bootable": True,
>>> > > > > >                     "ispublic": False,
>>> > > > > >                     "url": "{{ marvin_images_location 
>>> > > > > > }}/TinyCore-current.iso",
>>> > > > > >                     "ostype": 'Other Linux (64-bit)',
>>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
>>> > > > > >         },
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > I thought that we had come up with a solution for
>>> > download.cloud.com
>>> > > ,
>>> > > > by
>>> > > > > > having a mirrorlist hosted in Community Apache 'space' with
>>> anyone
>>> > > able
>>> > > > > to
>>> > > > > > out themselves forward as a mirror.
>>> > > > > > But I must admit I lost track of whether anyone made the
>>> requisite
>>> > > > > changes
>>> > > > > > in code....
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > Kind regards,
>>> > > > > >
>>> > > > > > Paul Angus
>>> > > > > >
>>> > > > > > paul.angus@shapeblue.com
>>> > > > > > www.shapeblue.com
>>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK 
>>> > > > > > @shapeblue
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > -----Original Message-----
>>> > > > > > From: williamstevens@gmail.com 
>>> > > > > > [mailto:williamstevens@gmail.com]
>>> > On
>>> > > > > > Behalf Of Will Stevens
>>> > > > > > Sent: 24 February 2017 16:30
>>> > > > > > To: dev@cloudstack.apache.org
>>> > > > > > Subject: Re: Modern template hosting
>>> > > > > >
>>> > > > > > this is a hard questions.  in general, we should be setting 
>>> > > > > > up a
>>> > > mirror
>>> > > > > on
>>> > > > > > some cloudstack/apache domain and then mirror to other 
>>> > > > > > provided
>>> > > > > templates.
>>> > > > > >
>>> > > > > > we MUST come up with a solution to deprecate 
>>> > > > > > 'download.cloud.com
>>> ',
>>> > > > that
>>> > > > > > is going to be going away any day now.
>>> > > > > >
>>> > > > > > i don't know the right way to solve this to be honest, but 
>>> > > > > > if you
>>> > > have
>>> > > > > > ideas, i am willing to help.
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > *Will STEVENS*
>>> > > > > > Lead Developer
>>> > > > > >
>>> > > > > > <https://goo.gl/NYZ8KK>
>>> > > > > >
>>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
>>> njohnson@ena.com
>>> > >
>>> > > > > wrote:
>>> > > > > >
>>> > > > > > > So not to re-open a can of worms, but I’m in a situation 
>>> > > > > > > where
>>> I
>>> > > need
>>> > > > > > > to come up with a Marvin component test that depends on a
>>> > template
>>> > > > > > > based on a kernel that’s relatively new, i.e., newer than
>>> Centos
>>> > > 5.3
>>> > > > /
>>> > > > > > Ubuntu 10.04 .
>>> > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 
>>> > > > > > > for
>>> KVM),
>>> > > but
>>> > > > > > > from looking at the thread "Migrating CloudStack content 
>>> > > > > > > from download.cloud.com” it looks like there is 
>>> > > > > > > resistance to using
>>> > > this
>>> > > > at
>>> > > > > > > least for hosting system vm templates over concerns of
>>> > neutrality.
>>> > > > > > > Would this be suitable for a component test?  If not, 
>>> > > > > > > what is a
>>> > > > > “blessed”
>>> > > > > > > template location?
>>> > > > > > >
>>> > > > > > > Thanks in advance!
>>> > > > > > >
>>> > > > > > > Nathan Johnson
>>> > > > > > > R&D Engineer
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville, TN 37211 
>>> > > > > > > General Office: 615-312-6000
>>> > > > > > >
>>> > > > > > > website | blog | support
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > DISCLAIMER
>>> > > > > ==========
>>> > > > > This e-mail may contain privileged and confidential 
>>> > > > > information
>>> which
>>> > > is
>>> > > > > the property of Accelerite, a Persistent Systems business. It 
>>> > > > > is
>>> > > intended
>>> > > > > only for the use of the individual or entity to which it is
>>> > addressed.
>>> > > If
>>> > > > > you are not the intended recipient, you are not authorized to 
>>> > > > > read,
>>> > > > retain,
>>> > > > > copy, print, distribute or use this message. If you have 
>>> > > > > received
>>> > this
>>> > > > > communication in error, please notify the sender and delete 
>>> > > > > all
>>> > copies
>>> > > of
>>> > > > > this message. Accelerite, a Persistent Systems business does 
>>> > > > > not
>>> > accept
>>> > > > any
>>> > > > > liability for virus infected mails.
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>
>abhinandan.prateek@shapeblue.com
>www.shapeblue.com
>53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>  
> 
>

abhinandan.prateek@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue

RE: Modern template hosting

Posted by Paul Angus <pa...@shapeblue.com>.

-1

We are trying to get MORE people to use CloudStack.  And raising the barrier to entry does nothing to help that.


Kind regards,

Paul Angus

paul.angus@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-----Original Message-----
From: Abhinandan Prateek [mailto:abhinandan.prateek@shapeblue.com] 
Sent: 01 March 2017 09:39
To: dev@cloudstack.apache.org
Subject: Re: Modern template hosting

+1

After seeing this discussion running in circles several times, I think we should at least get started with the simplest option.




On 27/02/17, 11:54 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:

>The alternative foolproof way is to simply not provide the automatic 
>download.
>
>Just document the mirror list URL where the templates can be found. I 
>don't know why people reject this option.
>
>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
>
>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens 
><wi...@gmail.com>
>wrote:
>
>> OK. Thanks for the heads up.
>>
>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:
>>
>> > Sounds workable. The downloader code in the SSVM won't follow 
>> > redirects I think.
>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d9cd0444
>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
>> > HttpTemplateDownloader.java#L93
>> > https://goo.gl/dSi0r5
>> >  Might need to add
>> > client.setRedirectStrategy(new LaxRedirectStrategy());
>> >
>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens 
>> > <ws...@cloudops.com>
>> > wrote:
>> >
>> > > We haven't opened a ticket yet because we don't have a strategy yet.
>> > >
>> > > What do you guys think of this:
>> > > - We setup a new github repo in the 'apache' org which consists 
>> > > of a
>> > single
>> > > file with a list of active/supported mirrors.
>> > > - I write a small web server, distributed as a binary, which can 
>> > > be
>> > hosted
>> > > by ASF Infra.  This web server will query the current list of 
>> > > mirrors
>> and
>> > > will select one and then do a 302 redirect to that mirror.
>> > >
>> > > The act of 'choosing' a mirror could be done in a number of ways.
>> > > - If we want to define an order, then it could just try from the 
>> > > top of
>> > the
>> > > list and work its way down.  It would curl the target to make 
>> > > sure it
>> > gets
>> > > a 200 and if it does, it would do a 302 redirect.
>> > > - Or, if we want to distribute the load across the mirrors, we 
>> > > could
>> pick
>> > > from the list randomly.  Again, doing a curl to verify the mirror 
>> > > is up
>> > and
>> > > then doing a redirect.
>> > > - If we want to get fancy, we could do a reverse IP lookup and 
>> > > try to
>> > match
>> > > the requester with their closest geographical mirror.
>> > >
>> > > Thoughts?
>> > >
>> > > *Will STEVENS*
>> > > Lead Developer
>> > >
>> > > <https://goo.gl/NYZ8KK>
>> > >
>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
>> chiradeepv@gmail.com
>> > >
>> > > wrote:
>> > >
>> > > > What steps are needed to set up a mirror? What does Infra need to do?
>> > Has
>> > > > anybody filed a ticket with Infra?
>> > > >
>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela < 
>> > > > raja.pullela@accelerite.com>
>> > > > wrote:
>> > > >
>> > > > > Hi will,
>> > > > >
>> > > > > I believe, we didn’t get to close ‘getting a mirror on Apache’
>> > because
>> > > we
>> > > > > needed someone on the Apache Infra side to close this. BTW, 
>> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has 
>> > > > > all/most of
>> > the
>> > > > > content.  Once we can close on the Apache mirror for hosting 
>> > > > > the
>> > > > content, I
>> > > > > can help assist getting the content there.
>> > > > >
>> > > > > For now, we have replicated the download.cloud.com content to 
>> > > > > ‘ s3.download.accelerite.com’.
>> > > > > Also, we are working on a set of steps/procedure to help with 
>> > > > > this change.  I will update everyone in about a week’s time 
>> > > > > on the
>> > details.
>> > > > >
>> > > > > Best,
>> > > > > Raja Pullela
>> > > > > Engineering Team,
>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA, 95054
>> > > > >
>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of 
>> > > > > Will Stevens" <williamstevens@gmail.com on behalf of
>> > wstevens@cloudops.com>
>> > > > > wrote:
>> > > > >
>> > > > > unfortunately the template mirror conversation got caught up 
>> > > > > in
>> > details
>> > > > and
>> > > > > nobody took the lead on implementing a solution.
>> > > > >
>> > > > > citrix has been pinging me every couple months to say 'dude, 
>> > > > > we
>> need
>> > to
>> > > > > remove the dependency on download.citrix.com', but i have not 
>> > > > > had
>> > the
>> > > > > cycles to get in and solve the problem.  the shutdown of that 
>> > > > > is
>> > > imminent
>> > > > > right now, so we need to solve it asap.
>> > > > >
>> > > > >
>> > > > >
>> > > > > *Will STEVENS*
>> > > > > Lead Developer
>> > > > >
>> > > > > <https://goo.gl/NYZ8KK>
>> > > > >
>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
>> > paul.angus@shapeblue.com
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Nathan,
>> > > > > >
>> > > > > > Ideally, if you put the template location in (or use a 
>> > > > > > template
>> > > defined
>> > > > > > in)  test_data.py then the actual location can be 
>> > > > > > overridden by
>> > > anyone
>> > > > > > testing.
>> > > > > >
>> > > > > > For Trillian, we've copied all of the templates that people 
>> > > > > > have
>> > > define
>> > > > > to
>> > > > > > a local repo and then replace the URLs in test_data.py to 
>> > > > > > reduce
>> > > > > bandwidth
>> > > > > > use and download times.
>> > > > > >
>> > > > > > Ie:
>> > > > > >
>> > > > > >             "bootableIso":
>> > > > > >                 {
>> > > > > >                     "displaytext": "Test Bootable ISO",
>> > > > > >                     "name": "testISO",
>> > > > > >                     "bootable": True,
>> > > > > >                     "ispublic": False,
>> > > > > >                     "url": "{{ marvin_images_location 
>> > > > > > }}/TinyCore-current.iso",
>> > > > > >                     "ostype": 'Other Linux (64-bit)',
>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
>> > > > > >         },
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > I thought that we had come up with a solution for
>> > download.cloud.com
>> > > ,
>> > > > by
>> > > > > > having a mirrorlist hosted in Community Apache 'space' with
>> anyone
>> > > able
>> > > > > to
>> > > > > > out themselves forward as a mirror.
>> > > > > > But I must admit I lost track of whether anyone made the
>> requisite
>> > > > > changes
>> > > > > > in code....
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > Kind regards,
>> > > > > >
>> > > > > > Paul Angus
>> > > > > >
>> > > > > > paul.angus@shapeblue.com
>> > > > > > www.shapeblue.com
>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK 
>> > > > > > @shapeblue
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > -----Original Message-----
>> > > > > > From: williamstevens@gmail.com 
>> > > > > > [mailto:williamstevens@gmail.com]
>> > On
>> > > > > > Behalf Of Will Stevens
>> > > > > > Sent: 24 February 2017 16:30
>> > > > > > To: dev@cloudstack.apache.org
>> > > > > > Subject: Re: Modern template hosting
>> > > > > >
>> > > > > > this is a hard questions.  in general, we should be setting 
>> > > > > > up a
>> > > mirror
>> > > > > on
>> > > > > > some cloudstack/apache domain and then mirror to other 
>> > > > > > provided
>> > > > > templates.
>> > > > > >
>> > > > > > we MUST come up with a solution to deprecate 
>> > > > > > 'download.cloud.com
>> ',
>> > > > that
>> > > > > > is going to be going away any day now.
>> > > > > >
>> > > > > > i don't know the right way to solve this to be honest, but 
>> > > > > > if you
>> > > have
>> > > > > > ideas, i am willing to help.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > *Will STEVENS*
>> > > > > > Lead Developer
>> > > > > >
>> > > > > > <https://goo.gl/NYZ8KK>
>> > > > > >
>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
>> njohnson@ena.com
>> > >
>> > > > > wrote:
>> > > > > >
>> > > > > > > So not to re-open a can of worms, but I’m in a situation 
>> > > > > > > where
>> I
>> > > need
>> > > > > > > to come up with a Marvin component test that depends on a
>> > template
>> > > > > > > based on a kernel that’s relatively new, i.e., newer than
>> Centos
>> > > 5.3
>> > > > /
>> > > > > > Ubuntu 10.04 .
>> > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 
>> > > > > > > for
>> KVM),
>> > > but
>> > > > > > > from looking at the thread "Migrating CloudStack content 
>> > > > > > > from download.cloud.com” it looks like there is 
>> > > > > > > resistance to using
>> > > this
>> > > > at
>> > > > > > > least for hosting system vm templates over concerns of
>> > neutrality.
>> > > > > > > Would this be suitable for a component test?  If not, 
>> > > > > > > what is a
>> > > > > “blessed”
>> > > > > > > template location?
>> > > > > > >
>> > > > > > > Thanks in advance!
>> > > > > > >
>> > > > > > > Nathan Johnson
>> > > > > > > R&D Engineer
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville, TN 37211 
>> > > > > > > General Office: 615-312-6000
>> > > > > > >
>> > > > > > > website | blog | support
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > DISCLAIMER
>> > > > > ==========
>> > > > > This e-mail may contain privileged and confidential 
>> > > > > information
>> which
>> > > is
>> > > > > the property of Accelerite, a Persistent Systems business. It 
>> > > > > is
>> > > intended
>> > > > > only for the use of the individual or entity to which it is
>> > addressed.
>> > > If
>> > > > > you are not the intended recipient, you are not authorized to 
>> > > > > read,
>> > > > retain,
>> > > > > copy, print, distribute or use this message. If you have 
>> > > > > received
>> > this
>> > > > > communication in error, please notify the sender and delete 
>> > > > > all
>> > copies
>> > > of
>> > > > > this message. Accelerite, a Persistent Systems business does 
>> > > > > not
>> > accept
>> > > > any
>> > > > > liability for virus infected mails.
>> > > > >
>> > > >
>> > >
>> >
>>

abhinandan.prateek@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue

Re: Modern template hosting

Posted by Abhinandan Prateek <ab...@shapeblue.com>.

+1

After seeing this discussion running in circles several times, I think we should at least get started with the simplest option.




On 27/02/17, 11:54 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:

>The alternative foolproof way is to simply not provide the automatic
>download.
>
>Just document the mirror list URL where the templates can be found. I don't
>know why people reject this option.
>
>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
>
>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <wi...@gmail.com>
>wrote:
>
>> OK. Thanks for the heads up.
>>
>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:
>>
>> > Sounds workable. The downloader code in the SSVM won't follow redirects I
>> > think.
>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d9cd0444a65a3
>> > 155fc9c921/core/src/com/cloud/storage/template/
>> > HttpTemplateDownloader.java#L93
>> > https://goo.gl/dSi0r5
>> >  Might need to add
>> > client.setRedirectStrategy(new LaxRedirectStrategy());
>> >
>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <ws...@cloudops.com>
>> > wrote:
>> >
>> > > We haven't opened a ticket yet because we don't have a strategy yet.
>> > >
>> > > What do you guys think of this:
>> > > - We setup a new github repo in the 'apache' org which consists of a
>> > single
>> > > file with a list of active/supported mirrors.
>> > > - I write a small web server, distributed as a binary, which can be
>> > hosted
>> > > by ASF Infra.  This web server will query the current list of mirrors
>> and
>> > > will select one and then do a 302 redirect to that mirror.
>> > >
>> > > The act of 'choosing' a mirror could be done in a number of ways.
>> > > - If we want to define an order, then it could just try from the top of
>> > the
>> > > list and work its way down.  It would curl the target to make sure it
>> > gets
>> > > a 200 and if it does, it would do a 302 redirect.
>> > > - Or, if we want to distribute the load across the mirrors, we could
>> pick
>> > > from the list randomly.  Again, doing a curl to verify the mirror is up
>> > and
>> > > then doing a redirect.
>> > > - If we want to get fancy, we could do a reverse IP lookup and try to
>> > match
>> > > the requester with their closest geographical mirror.
>> > >
>> > > Thoughts?
>> > >
>> > > *Will STEVENS*
>> > > Lead Developer
>> > >
>> > > <https://goo.gl/NYZ8KK>
>> > >
>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
>> chiradeepv@gmail.com
>> > >
>> > > wrote:
>> > >
>> > > > What steps are needed to set up a mirror? What does Infra need to do?
>> > Has
>> > > > anybody filed a ticket with Infra?
>> > > >
>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
>> > > > raja.pullela@accelerite.com>
>> > > > wrote:
>> > > >
>> > > > > Hi will,
>> > > > >
>> > > > > I believe, we didn’t get to close ‘getting a mirror on Apache’
>> > because
>> > > we
>> > > > > needed someone on the Apache Infra side to close this. BTW,
>> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has all/most of
>> > the
>> > > > > content.  Once we can close on the Apache mirror for hosting the
>> > > > content, I
>> > > > > can help assist getting the content there.
>> > > > >
>> > > > > For now, we have replicated the download.cloud.com content to ‘
>> > > > > s3.download.accelerite.com’.
>> > > > > Also, we are working on a set of steps/procedure to help with this
>> > > > > change.  I will update everyone in about a week’s time on the
>> > details.
>> > > > >
>> > > > > Best,
>> > > > > Raja Pullela
>> > > > > Engineering Team,
>> > > > > Accelerite, 2055 Laurelwood Road,
>> > > > > Santa Clara, CA, 95054
>> > > > >
>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of Will
>> > > > > Stevens" <williamstevens@gmail.com on behalf of
>> > wstevens@cloudops.com>
>> > > > > wrote:
>> > > > >
>> > > > > unfortunately the template mirror conversation got caught up in
>> > details
>> > > > and
>> > > > > nobody took the lead on implementing a solution.
>> > > > >
>> > > > > citrix has been pinging me every couple months to say 'dude, we
>> need
>> > to
>> > > > > remove the dependency on download.citrix.com', but i have not had
>> > the
>> > > > > cycles to get in and solve the problem.  the shutdown of that is
>> > > imminent
>> > > > > right now, so we need to solve it asap.
>> > > > >
>> > > > >
>> > > > >
>> > > > > *Will STEVENS*
>> > > > > Lead Developer
>> > > > >
>> > > > > <https://goo.gl/NYZ8KK>
>> > > > >
>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
>> > paul.angus@shapeblue.com
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Nathan,
>> > > > > >
>> > > > > > Ideally, if you put the template location in (or use a template
>> > > defined
>> > > > > > in)  test_data.py then the actual location can be overridden by
>> > > anyone
>> > > > > > testing.
>> > > > > >
>> > > > > > For Trillian, we've copied all of the templates that people have
>> > > define
>> > > > > to
>> > > > > > a local repo and then replace the URLs in test_data.py to reduce
>> > > > > bandwidth
>> > > > > > use and download times.
>> > > > > >
>> > > > > > Ie:
>> > > > > >
>> > > > > >             "bootableIso":
>> > > > > >                 {
>> > > > > >                     "displaytext": "Test Bootable ISO",
>> > > > > >                     "name": "testISO",
>> > > > > >                     "bootable": True,
>> > > > > >                     "ispublic": False,
>> > > > > >                     "url": "{{ marvin_images_location
>> > > > > > }}/TinyCore-current.iso",
>> > > > > >                     "ostype": 'Other Linux (64-bit)',
>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
>> > > > > >         },
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > I thought that we had come up with a solution for
>> > download.cloud.com
>> > > ,
>> > > > by
>> > > > > > having a mirrorlist hosted in Community Apache 'space' with
>> anyone
>> > > able
>> > > > > to
>> > > > > > out themselves forward as a mirror.
>> > > > > > But I must admit I lost track of whether anyone made the
>> requisite
>> > > > > changes
>> > > > > > in code....
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > Kind regards,
>> > > > > >
>> > > > > > Paul Angus
>> > > > > >
>> > > > > > paul.angus@shapeblue.com
>> > > > > > www.shapeblue.com
>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > > > > > @shapeblue
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > -----Original Message-----
>> > > > > > From: williamstevens@gmail.com [mailto:williamstevens@gmail.com]
>> > On
>> > > > > > Behalf Of Will Stevens
>> > > > > > Sent: 24 February 2017 16:30
>> > > > > > To: dev@cloudstack.apache.org
>> > > > > > Subject: Re: Modern template hosting
>> > > > > >
>> > > > > > this is a hard questions.  in general, we should be setting up a
>> > > mirror
>> > > > > on
>> > > > > > some cloudstack/apache domain and then mirror to other provided
>> > > > > templates.
>> > > > > >
>> > > > > > we MUST come up with a solution to deprecate 'download.cloud.com
>> ',
>> > > > that
>> > > > > > is going to be going away any day now.
>> > > > > >
>> > > > > > i don't know the right way to solve this to be honest, but if you
>> > > have
>> > > > > > ideas, i am willing to help.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > *Will STEVENS*
>> > > > > > Lead Developer
>> > > > > >
>> > > > > > <https://goo.gl/NYZ8KK>
>> > > > > >
>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
>> njohnson@ena.com
>> > >
>> > > > > wrote:
>> > > > > >
>> > > > > > > So not to re-open a can of worms, but I’m in a situation where
>> I
>> > > need
>> > > > > > > to come up with a Marvin component test that depends on a
>> > template
>> > > > > > > based on a kernel that’s relatively new, i.e., newer than
>> Centos
>> > > 5.3
>> > > > /
>> > > > > > Ubuntu 10.04 .
>> > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for
>> KVM),
>> > > but
>> > > > > > > from looking at the thread "Migrating CloudStack content from
>> > > > > > > download.cloud.com” it looks like there is resistance to using
>> > > this
>> > > > at
>> > > > > > > least for hosting system vm templates over concerns of
>> > neutrality.
>> > > > > > > Would this be suitable for a component test?  If not, what is a
>> > > > > “blessed”
>> > > > > > > template location?
>> > > > > > >
>> > > > > > > Thanks in advance!
>> > > > > > >
>> > > > > > > Nathan Johnson
>> > > > > > > R&D Engineer
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > 618 Grassmere Park Drive, Suite 12
>> > > > > > > Nashville, TN 37211
>> > > > > > > General Office: 615-312-6000
>> > > > > > >
>> > > > > > > website | blog | support
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > DISCLAIMER
>> > > > > ==========
>> > > > > This e-mail may contain privileged and confidential information
>> which
>> > > is
>> > > > > the property of Accelerite, a Persistent Systems business. It is
>> > > intended
>> > > > > only for the use of the individual or entity to which it is
>> > addressed.
>> > > If
>> > > > > you are not the intended recipient, you are not authorized to read,
>> > > > retain,
>> > > > > copy, print, distribute or use this message. If you have received
>> > this
>> > > > > communication in error, please notify the sender and delete all
>> > copies
>> > > of
>> > > > > this message. Accelerite, a Persistent Systems business does not
>> > accept
>> > > > any
>> > > > > liability for virus infected mails.
>> > > > >
>> > > >
>> > >
>> >
>>

abhinandan.prateek@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue

Re: Modern template hosting

Posted by Chiradeep Vittal <ch...@gmail.com>.

The alternative foolproof way is to simply not provide the automatic
download.

Just document the mirror list URL where the templates can be found. I don't
know why people reject this option.

https://shankerbalan.net/blog/seed-cloudstack-templates-offline/

On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <wi...@gmail.com>
wrote:

> OK. Thanks for the heads up.
>
> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:
>
> > Sounds workable. The downloader code in the SSVM won't follow redirects I
> > think.
> > https://github.com/apache/cloudstack/blob/5511065fc20787619d9cd0444a65a3
> > 155fc9c921/core/src/com/cloud/storage/template/
> > HttpTemplateDownloader.java#L93
> > https://goo.gl/dSi0r5
> >  Might need to add
> > client.setRedirectStrategy(new LaxRedirectStrategy());
> >
> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <ws...@cloudops.com>
> > wrote:
> >
> > > We haven't opened a ticket yet because we don't have a strategy yet.
> > >
> > > What do you guys think of this:
> > > - We setup a new github repo in the 'apache' org which consists of a
> > single
> > > file with a list of active/supported mirrors.
> > > - I write a small web server, distributed as a binary, which can be
> > hosted
> > > by ASF Infra.  This web server will query the current list of mirrors
> and
> > > will select one and then do a 302 redirect to that mirror.
> > >
> > > The act of 'choosing' a mirror could be done in a number of ways.
> > > - If we want to define an order, then it could just try from the top of
> > the
> > > list and work its way down.  It would curl the target to make sure it
> > gets
> > > a 200 and if it does, it would do a 302 redirect.
> > > - Or, if we want to distribute the load across the mirrors, we could
> pick
> > > from the list randomly.  Again, doing a curl to verify the mirror is up
> > and
> > > then doing a redirect.
> > > - If we want to get fancy, we could do a reverse IP lookup and try to
> > match
> > > the requester with their closest geographical mirror.
> > >
> > > Thoughts?
> > >
> > > *Will STEVENS*
> > > Lead Developer
> > >
> > > <https://goo.gl/NYZ8KK>
> > >
> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> chiradeepv@gmail.com
> > >
> > > wrote:
> > >
> > > > What steps are needed to set up a mirror? What does Infra need to do?
> > Has
> > > > anybody filed a ticket with Infra?
> > > >
> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > raja.pullela@accelerite.com>
> > > > wrote:
> > > >
> > > > > Hi will,
> > > > >
> > > > > I believe, we didn’t get to close ‘getting a mirror on Apache’
> > because
> > > we
> > > > > needed someone on the Apache Infra side to close this. BTW,
> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has all/most of
> > the
> > > > > content.  Once we can close on the Apache mirror for hosting the
> > > > content, I
> > > > > can help assist getting the content there.
> > > > >
> > > > > For now, we have replicated the download.cloud.com content to ‘
> > > > > s3.download.accelerite.com’.
> > > > > Also, we are working on a set of steps/procedure to help with this
> > > > > change.  I will update everyone in about a week’s time on the
> > details.
> > > > >
> > > > > Best,
> > > > > Raja Pullela
> > > > > Engineering Team,
> > > > > Accelerite, 2055 Laurelwood Road,
> > > > > Santa Clara, CA, 95054
> > > > >
> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of Will
> > > > > Stevens" <williamstevens@gmail.com on behalf of
> > wstevens@cloudops.com>
> > > > > wrote:
> > > > >
> > > > > unfortunately the template mirror conversation got caught up in
> > details
> > > > and
> > > > > nobody took the lead on implementing a solution.
> > > > >
> > > > > citrix has been pinging me every couple months to say 'dude, we
> need
> > to
> > > > > remove the dependency on download.citrix.com', but i have not had
> > the
> > > > > cycles to get in and solve the problem.  the shutdown of that is
> > > imminent
> > > > > right now, so we need to solve it asap.
> > > > >
> > > > >
> > > > >
> > > > > *Will STEVENS*
> > > > > Lead Developer
> > > > >
> > > > > <https://goo.gl/NYZ8KK>
> > > > >
> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > paul.angus@shapeblue.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Nathan,
> > > > > >
> > > > > > Ideally, if you put the template location in (or use a template
> > > defined
> > > > > > in)  test_data.py then the actual location can be overridden by
> > > anyone
> > > > > > testing.
> > > > > >
> > > > > > For Trillian, we've copied all of the templates that people have
> > > define
> > > > > to
> > > > > > a local repo and then replace the URLs in test_data.py to reduce
> > > > > bandwidth
> > > > > > use and download times.
> > > > > >
> > > > > > Ie:
> > > > > >
> > > > > >             "bootableIso":
> > > > > >                 {
> > > > > >                     "displaytext": "Test Bootable ISO",
> > > > > >                     "name": "testISO",
> > > > > >                     "bootable": True,
> > > > > >                     "ispublic": False,
> > > > > >                     "url": "{{ marvin_images_location
> > > > > > }}/TinyCore-current.iso",
> > > > > >                     "ostype": 'Other Linux (64-bit)',
> > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > >         },
> > > > > >
> > > > > >
> > > > > >
> > > > > > I thought that we had come up with a solution for
> > download.cloud.com
> > > ,
> > > > by
> > > > > > having a mirrorlist hosted in Community Apache 'space' with
> anyone
> > > able
> > > > > to
> > > > > > out themselves forward as a mirror.
> > > > > > But I must admit I lost track of whether anyone made the
> requisite
> > > > > changes
> > > > > > in code....
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Kind regards,
> > > > > >
> > > > > > Paul Angus
> > > > > >
> > > > > > paul.angus@shapeblue.com
> > > > > > www.shapeblue.com
> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > @shapeblue
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: williamstevens@gmail.com [mailto:williamstevens@gmail.com]
> > On
> > > > > > Behalf Of Will Stevens
> > > > > > Sent: 24 February 2017 16:30
> > > > > > To: dev@cloudstack.apache.org
> > > > > > Subject: Re: Modern template hosting
> > > > > >
> > > > > > this is a hard questions.  in general, we should be setting up a
> > > mirror
> > > > > on
> > > > > > some cloudstack/apache domain and then mirror to other provided
> > > > > templates.
> > > > > >
> > > > > > we MUST come up with a solution to deprecate 'download.cloud.com
> ',
> > > > that
> > > > > > is going to be going away any day now.
> > > > > >
> > > > > > i don't know the right way to solve this to be honest, but if you
> > > have
> > > > > > ideas, i am willing to help.
> > > > > >
> > > > > >
> > > > > >
> > > > > > *Will STEVENS*
> > > > > > Lead Developer
> > > > > >
> > > > > > <https://goo.gl/NYZ8KK>
> > > > > >
> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> njohnson@ena.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > So not to re-open a can of worms, but I’m in a situation where
> I
> > > need
> > > > > > > to come up with a Marvin component test that depends on a
> > template
> > > > > > > based on a kernel that’s relatively new, i.e., newer than
> Centos
> > > 5.3
> > > > /
> > > > > > Ubuntu 10.04 .
> > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for
> KVM),
> > > but
> > > > > > > from looking at the thread "Migrating CloudStack content from
> > > > > > > download.cloud.com” it looks like there is resistance to using
> > > this
> > > > at
> > > > > > > least for hosting system vm templates over concerns of
> > neutrality.
> > > > > > > Would this be suitable for a component test?  If not, what is a
> > > > > “blessed”
> > > > > > > template location?
> > > > > > >
> > > > > > > Thanks in advance!
> > > > > > >
> > > > > > > Nathan Johnson
> > > > > > > R&D Engineer
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > Nashville, TN 37211
> > > > > > > General Office: 615-312-6000
> > > > > > >
> > > > > > > website | blog | support
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > DISCLAIMER
> > > > > ==========
> > > > > This e-mail may contain privileged and confidential information
> which
> > > is
> > > > > the property of Accelerite, a Persistent Systems business. It is
> > > intended
> > > > > only for the use of the individual or entity to which it is
> > addressed.
> > > If
> > > > > you are not the intended recipient, you are not authorized to read,
> > > > retain,
> > > > > copy, print, distribute or use this message. If you have received
> > this
> > > > > communication in error, please notify the sender and delete all
> > copies
> > > of
> > > > > this message. Accelerite, a Persistent Systems business does not
> > accept
> > > > any
> > > > > liability for virus infected mails.
> > > > >
> > > >
> > >
> >
>

Re: Modern template hosting

Posted by Will Stevens <wi...@gmail.com>.

OK. Thanks for the heads up.

On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <ch...@gmail.com> wrote:

> Sounds workable. The downloader code in the SSVM won't follow redirects I
> think.
> https://github.com/apache/cloudstack/blob/5511065fc20787619d9cd0444a65a3
> 155fc9c921/core/src/com/cloud/storage/template/
> HttpTemplateDownloader.java#L93
> https://goo.gl/dSi0r5
>  Might need to add
> client.setRedirectStrategy(new LaxRedirectStrategy());
>
> On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <ws...@cloudops.com>
> wrote:
>
> > We haven't opened a ticket yet because we don't have a strategy yet.
> >
> > What do you guys think of this:
> > - We setup a new github repo in the 'apache' org which consists of a
> single
> > file with a list of active/supported mirrors.
> > - I write a small web server, distributed as a binary, which can be
> hosted
> > by ASF Infra.  This web server will query the current list of mirrors and
> > will select one and then do a 302 redirect to that mirror.
> >
> > The act of 'choosing' a mirror could be done in a number of ways.
> > - If we want to define an order, then it could just try from the top of
> the
> > list and work its way down.  It would curl the target to make sure it
> gets
> > a 200 and if it does, it would do a 302 redirect.
> > - Or, if we want to distribute the load across the mirrors, we could pick
> > from the list randomly.  Again, doing a curl to verify the mirror is up
> and
> > then doing a redirect.
> > - If we want to get fancy, we could do a reverse IP lookup and try to
> match
> > the requester with their closest geographical mirror.
> >
> > Thoughts?
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <chiradeepv@gmail.com
> >
> > wrote:
> >
> > > What steps are needed to set up a mirror? What does Infra need to do?
> Has
> > > anybody filed a ticket with Infra?
> > >
> > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > raja.pullela@accelerite.com>
> > > wrote:
> > >
> > > > Hi will,
> > > >
> > > > I believe, we didn’t get to close ‘getting a mirror on Apache’
> because
> > we
> > > > needed someone on the Apache Infra side to close this. BTW,
> > > > cloudstack-apt.get.eu (I think Nux manages this?) has all/most of
> the
> > > > content.  Once we can close on the Apache mirror for hosting the
> > > content, I
> > > > can help assist getting the content there.
> > > >
> > > > For now, we have replicated the download.cloud.com content to ‘
> > > > s3.download.accelerite.com’.
> > > > Also, we are working on a set of steps/procedure to help with this
> > > > change.  I will update everyone in about a week’s time on the
> details.
> > > >
> > > > Best,
> > > > Raja Pullela
> > > > Engineering Team,
> > > > Accelerite, 2055 Laurelwood Road,
> > > > Santa Clara, CA, 95054
> > > >
> > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of Will
> > > > Stevens" <williamstevens@gmail.com on behalf of
> wstevens@cloudops.com>
> > > > wrote:
> > > >
> > > > unfortunately the template mirror conversation got caught up in
> details
> > > and
> > > > nobody took the lead on implementing a solution.
> > > >
> > > > citrix has been pinging me every couple months to say 'dude, we need
> to
> > > > remove the dependency on download.citrix.com', but i have not had
> the
> > > > cycles to get in and solve the problem.  the shutdown of that is
> > imminent
> > > > right now, so we need to solve it asap.
> > > >
> > > >
> > > >
> > > > *Will STEVENS*
> > > > Lead Developer
> > > >
> > > > <https://goo.gl/NYZ8KK>
> > > >
> > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> paul.angus@shapeblue.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Nathan,
> > > > >
> > > > > Ideally, if you put the template location in (or use a template
> > defined
> > > > > in)  test_data.py then the actual location can be overridden by
> > anyone
> > > > > testing.
> > > > >
> > > > > For Trillian, we've copied all of the templates that people have
> > define
> > > > to
> > > > > a local repo and then replace the URLs in test_data.py to reduce
> > > > bandwidth
> > > > > use and download times.
> > > > >
> > > > > Ie:
> > > > >
> > > > >             "bootableIso":
> > > > >                 {
> > > > >                     "displaytext": "Test Bootable ISO",
> > > > >                     "name": "testISO",
> > > > >                     "bootable": True,
> > > > >                     "ispublic": False,
> > > > >                     "url": "{{ marvin_images_location
> > > > > }}/TinyCore-current.iso",
> > > > >                     "ostype": 'Other Linux (64-bit)',
> > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > >         },
> > > > >
> > > > >
> > > > >
> > > > > I thought that we had come up with a solution for
> download.cloud.com
> > ,
> > > by
> > > > > having a mirrorlist hosted in Community Apache 'space' with anyone
> > able
> > > > to
> > > > > out themselves forward as a mirror.
> > > > > But I must admit I lost track of whether anyone made the requisite
> > > > changes
> > > > > in code....
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Kind regards,
> > > > >
> > > > > Paul Angus
> > > > >
> > > > > paul.angus@shapeblue.com
> > > > > www.shapeblue.com
> > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > @shapeblue
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: williamstevens@gmail.com [mailto:williamstevens@gmail.com]
> On
> > > > > Behalf Of Will Stevens
> > > > > Sent: 24 February 2017 16:30
> > > > > To: dev@cloudstack.apache.org
> > > > > Subject: Re: Modern template hosting
> > > > >
> > > > > this is a hard questions.  in general, we should be setting up a
> > mirror
> > > > on
> > > > > some cloudstack/apache domain and then mirror to other provided
> > > > templates.
> > > > >
> > > > > we MUST come up with a solution to deprecate 'download.cloud.com',
> > > that
> > > > > is going to be going away any day now.
> > > > >
> > > > > i don't know the right way to solve this to be honest, but if you
> > have
> > > > > ideas, i am willing to help.
> > > > >
> > > > >
> > > > >
> > > > > *Will STEVENS*
> > > > > Lead Developer
> > > > >
> > > > > <https://goo.gl/NYZ8KK>
> > > > >
> > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <njohnson@ena.com
> >
> > > > wrote:
> > > > >
> > > > > > So not to re-open a can of worms, but I’m in a situation where I
> > need
> > > > > > to come up with a Marvin component test that depends on a
> template
> > > > > > based on a kernel that’s relatively new, i.e., newer than Centos
> > 5.3
> > > /
> > > > > Ubuntu 10.04 .
> > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM),
> > but
> > > > > > from looking at the thread "Migrating CloudStack content from
> > > > > > download.cloud.com” it looks like there is resistance to using
> > this
> > > at
> > > > > > least for hosting system vm templates over concerns of
> neutrality.
> > > > > > Would this be suitable for a component test?  If not, what is a
> > > > “blessed”
> > > > > > template location?
> > > > > >
> > > > > > Thanks in advance!
> > > > > >
> > > > > > Nathan Johnson
> > > > > > R&D Engineer
> > > > > >
> > > > > >
> > > > > >
> > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > Nashville, TN 37211
> > > > > > General Office: 615-312-6000
> > > > > >
> > > > > > website | blog | support
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > DISCLAIMER
> > > > ==========
> > > > This e-mail may contain privileged and confidential information which
> > is
> > > > the property of Accelerite, a Persistent Systems business. It is
> > intended
> > > > only for the use of the individual or entity to which it is
> addressed.
> > If
> > > > you are not the intended recipient, you are not authorized to read,
> > > retain,
> > > > copy, print, distribute or use this message. If you have received
> this
> > > > communication in error, please notify the sender and delete all
> copies
> > of
> > > > this message. Accelerite, a Persistent Systems business does not
> accept
> > > any
> > > > liability for virus infected mails.
> > > >
> > >
> >
>

Re: Modern template hosting

Posted by Chiradeep Vittal <ch...@gmail.com>.

Sounds workable. The downloader code in the SSVM won't follow redirects I
think.
https://github.com/apache/cloudstack/blob/5511065fc20787619d9cd0444a65a3155fc9c921/core/src/com/cloud/storage/template/HttpTemplateDownloader.java#L93
https://goo.gl/dSi0r5
 Might need to add
client.setRedirectStrategy(new LaxRedirectStrategy());

On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <ws...@cloudops.com> wrote:

> We haven't opened a ticket yet because we don't have a strategy yet.
>
> What do you guys think of this:
> - We setup a new github repo in the 'apache' org which consists of a single
> file with a list of active/supported mirrors.
> - I write a small web server, distributed as a binary, which can be hosted
> by ASF Infra.  This web server will query the current list of mirrors and
> will select one and then do a 302 redirect to that mirror.
>
> The act of 'choosing' a mirror could be done in a number of ways.
> - If we want to define an order, then it could just try from the top of the
> list and work its way down.  It would curl the target to make sure it gets
> a 200 and if it does, it would do a 302 redirect.
> - Or, if we want to distribute the load across the mirrors, we could pick
> from the list randomly.  Again, doing a curl to verify the mirror is up and
> then doing a redirect.
> - If we want to get fancy, we could do a reverse IP lookup and try to match
> the requester with their closest geographical mirror.
>
> Thoughts?
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <ch...@gmail.com>
> wrote:
>
> > What steps are needed to set up a mirror? What does Infra need to do? Has
> > anybody filed a ticket with Infra?
> >
> > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > raja.pullela@accelerite.com>
> > wrote:
> >
> > > Hi will,
> > >
> > > I believe, we didn’t get to close ‘getting a mirror on Apache’ because
> we
> > > needed someone on the Apache Infra side to close this. BTW,
> > > cloudstack-apt.get.eu (I think Nux manages this?) has all/most of the
> > > content.  Once we can close on the Apache mirror for hosting the
> > content, I
> > > can help assist getting the content there.
> > >
> > > For now, we have replicated the download.cloud.com content to ‘
> > > s3.download.accelerite.com’.
> > > Also, we are working on a set of steps/procedure to help with this
> > > change.  I will update everyone in about a week’s time on the details.
> > >
> > > Best,
> > > Raja Pullela
> > > Engineering Team,
> > > Accelerite, 2055 Laurelwood Road,
> > > Santa Clara, CA, 95054
> > >
> > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of Will
> > > Stevens" <williamstevens@gmail.com on behalf of wstevens@cloudops.com>
> > > wrote:
> > >
> > > unfortunately the template mirror conversation got caught up in details
> > and
> > > nobody took the lead on implementing a solution.
> > >
> > > citrix has been pinging me every couple months to say 'dude, we need to
> > > remove the dependency on download.citrix.com', but i have not had the
> > > cycles to get in and solve the problem.  the shutdown of that is
> imminent
> > > right now, so we need to solve it asap.
> > >
> > >
> > >
> > > *Will STEVENS*
> > > Lead Developer
> > >
> > > <https://goo.gl/NYZ8KK>
> > >
> > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <paul.angus@shapeblue.com
> >
> > > wrote:
> > >
> > > > Hi Nathan,
> > > >
> > > > Ideally, if you put the template location in (or use a template
> defined
> > > > in)  test_data.py then the actual location can be overridden by
> anyone
> > > > testing.
> > > >
> > > > For Trillian, we've copied all of the templates that people have
> define
> > > to
> > > > a local repo and then replace the URLs in test_data.py to reduce
> > > bandwidth
> > > > use and download times.
> > > >
> > > > Ie:
> > > >
> > > >             "bootableIso":
> > > >                 {
> > > >                     "displaytext": "Test Bootable ISO",
> > > >                     "name": "testISO",
> > > >                     "bootable": True,
> > > >                     "ispublic": False,
> > > >                     "url": "{{ marvin_images_location
> > > > }}/TinyCore-current.iso",
> > > >                     "ostype": 'Other Linux (64-bit)',
> > > >                     "mode": 'HTTP_DOWNLOAD'
> > > >         },
> > > >
> > > >
> > > >
> > > > I thought that we had come up with a solution for download.cloud.com
> ,
> > by
> > > > having a mirrorlist hosted in Community Apache 'space' with anyone
> able
> > > to
> > > > out themselves forward as a mirror.
> > > > But I must admit I lost track of whether anyone made the requisite
> > > changes
> > > > in code....
> > > >
> > > >
> > > >
> > > >
> > > > Kind regards,
> > > >
> > > > Paul Angus
> > > >
> > > > paul.angus@shapeblue.com
> > > > www.shapeblue.com
> > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > @shapeblue
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
> > > > Behalf Of Will Stevens
> > > > Sent: 24 February 2017 16:30
> > > > To: dev@cloudstack.apache.org
> > > > Subject: Re: Modern template hosting
> > > >
> > > > this is a hard questions.  in general, we should be setting up a
> mirror
> > > on
> > > > some cloudstack/apache domain and then mirror to other provided
> > > templates.
> > > >
> > > > we MUST come up with a solution to deprecate 'download.cloud.com',
> > that
> > > > is going to be going away any day now.
> > > >
> > > > i don't know the right way to solve this to be honest, but if you
> have
> > > > ideas, i am willing to help.
> > > >
> > > >
> > > >
> > > > *Will STEVENS*
> > > > Lead Developer
> > > >
> > > > <https://goo.gl/NYZ8KK>
> > > >
> > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <nj...@ena.com>
> > > wrote:
> > > >
> > > > > So not to re-open a can of worms, but I’m in a situation where I
> need
> > > > > to come up with a Marvin component test that depends on a template
> > > > > based on a kernel that’s relatively new, i.e., newer than Centos
> 5.3
> > /
> > > > Ubuntu 10.04 .
> > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM),
> but
> > > > > from looking at the thread "Migrating CloudStack content from
> > > > > download.cloud.com” it looks like there is resistance to using
> this
> > at
> > > > > least for hosting system vm templates over concerns of neutrality.
> > > > > Would this be suitable for a component test?  If not, what is a
> > > “blessed”
> > > > > template location?
> > > > >
> > > > > Thanks in advance!
> > > > >
> > > > > Nathan Johnson
> > > > > R&D Engineer
> > > > >
> > > > >
> > > > >
> > > > > 618 Grassmere Park Drive, Suite 12
> > > > > Nashville, TN 37211
> > > > > General Office: 615-312-6000
> > > > >
> > > > > website | blog | support
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > DISCLAIMER
> > > ==========
> > > This e-mail may contain privileged and confidential information which
> is
> > > the property of Accelerite, a Persistent Systems business. It is
> intended
> > > only for the use of the individual or entity to which it is addressed.
> If
> > > you are not the intended recipient, you are not authorized to read,
> > retain,
> > > copy, print, distribute or use this message. If you have received this
> > > communication in error, please notify the sender and delete all copies
> of
> > > this message. Accelerite, a Persistent Systems business does not accept
> > any
> > > liability for virus infected mails.
> > >
> >
>

Re: Modern template hosting

Posted by Will Stevens <ws...@cloudops.com>.

We haven't opened a ticket yet because we don't have a strategy yet.

What do you guys think of this:
- We setup a new github repo in the 'apache' org which consists of a single
file with a list of active/supported mirrors.
- I write a small web server, distributed as a binary, which can be hosted
by ASF Infra.  This web server will query the current list of mirrors and
will select one and then do a 302 redirect to that mirror.

The act of 'choosing' a mirror could be done in a number of ways.
- If we want to define an order, then it could just try from the top of the
list and work its way down.  It would curl the target to make sure it gets
a 200 and if it does, it would do a 302 redirect.
- Or, if we want to distribute the load across the mirrors, we could pick
from the list randomly.  Again, doing a curl to verify the mirror is up and
then doing a redirect.
- If we want to get fancy, we could do a reverse IP lookup and try to match
the requester with their closest geographical mirror.

Thoughts?

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <ch...@gmail.com>
wrote:

> What steps are needed to set up a mirror? What does Infra need to do? Has
> anybody filed a ticket with Infra?
>
> On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> raja.pullela@accelerite.com>
> wrote:
>
> > Hi will,
> >
> > I believe, we didn’t get to close ‘getting a mirror on Apache’ because we
> > needed someone on the Apache Infra side to close this. BTW,
> > cloudstack-apt.get.eu (I think Nux manages this?) has all/most of the
> > content.  Once we can close on the Apache mirror for hosting the
> content, I
> > can help assist getting the content there.
> >
> > For now, we have replicated the download.cloud.com content to ‘
> > s3.download.accelerite.com’.
> > Also, we are working on a set of steps/procedure to help with this
> > change.  I will update everyone in about a week’s time on the details.
> >
> > Best,
> > Raja Pullela
> > Engineering Team,
> > Accelerite, 2055 Laurelwood Road,
> > Santa Clara, CA, 95054
> >
> > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of Will
> > Stevens" <williamstevens@gmail.com on behalf of wstevens@cloudops.com>
> > wrote:
> >
> > unfortunately the template mirror conversation got caught up in details
> and
> > nobody took the lead on implementing a solution.
> >
> > citrix has been pinging me every couple months to say 'dude, we need to
> > remove the dependency on download.citrix.com', but i have not had the
> > cycles to get in and solve the problem.  the shutdown of that is imminent
> > right now, so we need to solve it asap.
> >
> >
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <pa...@shapeblue.com>
> > wrote:
> >
> > > Hi Nathan,
> > >
> > > Ideally, if you put the template location in (or use a template defined
> > > in)  test_data.py then the actual location can be overridden by anyone
> > > testing.
> > >
> > > For Trillian, we've copied all of the templates that people have define
> > to
> > > a local repo and then replace the URLs in test_data.py to reduce
> > bandwidth
> > > use and download times.
> > >
> > > Ie:
> > >
> > >             "bootableIso":
> > >                 {
> > >                     "displaytext": "Test Bootable ISO",
> > >                     "name": "testISO",
> > >                     "bootable": True,
> > >                     "ispublic": False,
> > >                     "url": "{{ marvin_images_location
> > > }}/TinyCore-current.iso",
> > >                     "ostype": 'Other Linux (64-bit)',
> > >                     "mode": 'HTTP_DOWNLOAD'
> > >         },
> > >
> > >
> > >
> > > I thought that we had come up with a solution for download.cloud.com,
> by
> > > having a mirrorlist hosted in Community Apache 'space' with anyone able
> > to
> > > out themselves forward as a mirror.
> > > But I must admit I lost track of whether anyone made the requisite
> > changes
> > > in code....
> > >
> > >
> > >
> > >
> > > Kind regards,
> > >
> > > Paul Angus
> > >
> > > paul.angus@shapeblue.com
> > > www.shapeblue.com
> > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > @shapeblue
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
> > > Behalf Of Will Stevens
> > > Sent: 24 February 2017 16:30
> > > To: dev@cloudstack.apache.org
> > > Subject: Re: Modern template hosting
> > >
> > > this is a hard questions.  in general, we should be setting up a mirror
> > on
> > > some cloudstack/apache domain and then mirror to other provided
> > templates.
> > >
> > > we MUST come up with a solution to deprecate 'download.cloud.com',
> that
> > > is going to be going away any day now.
> > >
> > > i don't know the right way to solve this to be honest, but if you have
> > > ideas, i am willing to help.
> > >
> > >
> > >
> > > *Will STEVENS*
> > > Lead Developer
> > >
> > > <https://goo.gl/NYZ8KK>
> > >
> > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <nj...@ena.com>
> > wrote:
> > >
> > > > So not to re-open a can of worms, but I’m in a situation where I need
> > > > to come up with a Marvin component test that depends on a template
> > > > based on a kernel that’s relatively new, i.e., newer than Centos 5.3
> /
> > > Ubuntu 10.04 .
> > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
> > > > from looking at the thread "Migrating CloudStack content from
> > > > download.cloud.com” it looks like there is resistance to using this
> at
> > > > least for hosting system vm templates over concerns of neutrality.
> > > > Would this be suitable for a component test?  If not, what is a
> > “blessed”
> > > > template location?
> > > >
> > > > Thanks in advance!
> > > >
> > > > Nathan Johnson
> > > > R&D Engineer
> > > >
> > > >
> > > >
> > > > 618 Grassmere Park Drive, Suite 12
> > > > Nashville, TN 37211
> > > > General Office: 615-312-6000
> > > >
> > > > website | blog | support
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
> >
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information which is
> > the property of Accelerite, a Persistent Systems business. It is intended
> > only for the use of the individual or entity to which it is addressed. If
> > you are not the intended recipient, you are not authorized to read,
> retain,
> > copy, print, distribute or use this message. If you have received this
> > communication in error, please notify the sender and delete all copies of
> > this message. Accelerite, a Persistent Systems business does not accept
> any
> > liability for virus infected mails.
> >
>

Re: Modern template hosting

Posted by Chiradeep Vittal <ch...@gmail.com>.

What steps are needed to set up a mirror? What does Infra need to do? Has
anybody filed a ticket with Infra?

On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <ra...@accelerite.com>
wrote:

> Hi will,
>
> I believe, we didn’t get to close ‘getting a mirror on Apache’ because we
> needed someone on the Apache Infra side to close this. BTW,
> cloudstack-apt.get.eu (I think Nux manages this?) has all/most of the
> content.  Once we can close on the Apache mirror for hosting the content, I
> can help assist getting the content there.
>
> For now, we have replicated the download.cloud.com content to ‘
> s3.download.accelerite.com’.
> Also, we are working on a set of steps/procedure to help with this
> change.  I will update everyone in about a week’s time on the details.
>
> Best,
> Raja Pullela
> Engineering Team,
> Accelerite, 2055 Laurelwood Road,
> Santa Clara, CA, 95054
>
> On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of Will
> Stevens" <williamstevens@gmail.com on behalf of wstevens@cloudops.com>
> wrote:
>
> unfortunately the template mirror conversation got caught up in details and
> nobody took the lead on implementing a solution.
>
> citrix has been pinging me every couple months to say 'dude, we need to
> remove the dependency on download.citrix.com', but i have not had the
> cycles to get in and solve the problem.  the shutdown of that is imminent
> right now, so we need to solve it asap.
>
>
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <pa...@shapeblue.com>
> wrote:
>
> > Hi Nathan,
> >
> > Ideally, if you put the template location in (or use a template defined
> > in)  test_data.py then the actual location can be overridden by anyone
> > testing.
> >
> > For Trillian, we've copied all of the templates that people have define
> to
> > a local repo and then replace the URLs in test_data.py to reduce
> bandwidth
> > use and download times.
> >
> > Ie:
> >
> >             "bootableIso":
> >                 {
> >                     "displaytext": "Test Bootable ISO",
> >                     "name": "testISO",
> >                     "bootable": True,
> >                     "ispublic": False,
> >                     "url": "{{ marvin_images_location
> > }}/TinyCore-current.iso",
> >                     "ostype": 'Other Linux (64-bit)',
> >                     "mode": 'HTTP_DOWNLOAD'
> >         },
> >
> >
> >
> > I thought that we had come up with a solution for download.cloud.com, by
> > having a mirrorlist hosted in Community Apache 'space' with anyone able
> to
> > out themselves forward as a mirror.
> > But I must admit I lost track of whether anyone made the requisite
> changes
> > in code....
> >
> >
> >
> >
> > Kind regards,
> >
> > Paul Angus
> >
> > paul.angus@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> > -----Original Message-----
> > From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
> > Behalf Of Will Stevens
> > Sent: 24 February 2017 16:30
> > To: dev@cloudstack.apache.org
> > Subject: Re: Modern template hosting
> >
> > this is a hard questions.  in general, we should be setting up a mirror
> on
> > some cloudstack/apache domain and then mirror to other provided
> templates.
> >
> > we MUST come up with a solution to deprecate 'download.cloud.com', that
> > is going to be going away any day now.
> >
> > i don't know the right way to solve this to be honest, but if you have
> > ideas, i am willing to help.
> >
> >
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <nj...@ena.com>
> wrote:
> >
> > > So not to re-open a can of worms, but I’m in a situation where I need
> > > to come up with a Marvin component test that depends on a template
> > > based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
> > Ubuntu 10.04 .
> > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
> > > from looking at the thread "Migrating CloudStack content from
> > > download.cloud.com” it looks like there is resistance to using this at
> > > least for hosting system vm templates over concerns of neutrality.
> > > Would this be suitable for a component test?  If not, what is a
> “blessed”
> > > template location?
> > >
> > > Thanks in advance!
> > >
> > > Nathan Johnson
> > > R&D Engineer
> > >
> > >
> > >
> > > 618 Grassmere Park Drive, Suite 12
> > > Nashville, TN 37211
> > > General Office: 615-312-6000
> > >
> > > website | blog | support
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
>
>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Accelerite, a Persistent Systems business. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Accelerite, a Persistent Systems business does not accept any
> liability for virus infected mails.
>

Re: Modern template hosting

Posted by Raja Pullela <ra...@accelerite.com>.

Hi will, 

I believe, we didn’t get to close ‘getting a mirror on Apache’ because we needed someone on the Apache Infra side to close this. BTW, cloudstack-apt.get.eu (I think Nux manages this?) has all/most of the content.  Once we can close on the Apache mirror for hosting the content, I can help assist getting the content there.

For now, we have replicated the download.cloud.com content to ‘s3.download.accelerite.com’.
Also, we are working on a set of steps/procedure to help with this change.  I will update everyone in about a week’s time on the details.

Best,
Raja Pullela
Engineering Team,
Accelerite, 2055 Laurelwood Road,
Santa Clara, CA, 95054

On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of Will Stevens" <williamstevens@gmail.com on behalf of wstevens@cloudops.com> wrote:

unfortunately the template mirror conversation got caught up in details and
nobody took the lead on implementing a solution.

citrix has been pinging me every couple months to say 'dude, we need to
remove the dependency on download.citrix.com', but i have not had the
cycles to get in and solve the problem.  the shutdown of that is imminent
right now, so we need to solve it asap.

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <pa...@shapeblue.com>
wrote:

> Hi Nathan,
>
> Ideally, if you put the template location in (or use a template defined
> in)  test_data.py then the actual location can be overridden by anyone
> testing.
>
> For Trillian, we've copied all of the templates that people have define to
> a local repo and then replace the URLs in test_data.py to reduce bandwidth
> use and download times.
>
> Ie:
>
>             "bootableIso":
>                 {
>                     "displaytext": "Test Bootable ISO",
>                     "name": "testISO",
>                     "bootable": True,
>                     "ispublic": False,
>                     "url": "{{ marvin_images_location
> }}/TinyCore-current.iso",
>                     "ostype": 'Other Linux (64-bit)',
>                     "mode": 'HTTP_DOWNLOAD'
>         },
>
>
>
> I thought that we had come up with a solution for download.cloud.com, by
> having a mirrorlist hosted in Community Apache 'space' with anyone able to
> out themselves forward as a mirror.
> But I must admit I lost track of whether anyone made the requisite changes
> in code....
>
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
> Behalf Of Will Stevens
> Sent: 24 February 2017 16:30
> To: dev@cloudstack.apache.org
> Subject: Re: Modern template hosting
>
> this is a hard questions.  in general, we should be setting up a mirror on
> some cloudstack/apache domain and then mirror to other provided templates.
>
> we MUST come up with a solution to deprecate 'download.cloud.com', that
> is going to be going away any day now.
>
> i don't know the right way to solve this to be honest, but if you have
> ideas, i am willing to help.
>
>
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <nj...@ena.com> wrote:
>
> > So not to re-open a can of worms, but I’m in a situation where I need
> > to come up with a Marvin component test that depends on a template
> > based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
> Ubuntu 10.04 .
> > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
> > from looking at the thread "Migrating CloudStack content from
> > download.cloud.com” it looks like there is resistance to using this at
> > least for hosting system vm templates over concerns of neutrality.
> > Would this be suitable for a component test?  If not, what is a “blessed”
> > template location?
> >
> > Thanks in advance!
> >
> > Nathan Johnson
> > R&D Engineer
> >
> >
> >
> > 618 Grassmere Park Drive, Suite 12
> > Nashville, TN 37211
> > General Office: 615-312-6000
> >
> > website | blog | support
> >
> >
> >
> >
> >
> >
> >
>

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.

Re: Modern template hosting

Posted by Nux! <nu...@li.nux.ro>.

Chiradeep,

I wouldn't know, needs investigating. Could be that.

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Chiradeep Vittal" <ch...@gmail.com>
> To: "dev" <de...@cloudstack.apache.org>
> Sent: Sunday, 26 February, 2017 00:07:53
> Subject: Re: Modern template hosting

> Is the issue that ACS doesn't support 169.254.169.254 as the metadata URL?
> 
> For those hypervisors that use the Linux bridge and iptables, it should be easy
> to add a DNAT rule to translate it to the VR IP? I'm talking about code change
> here.
> Or send down a static route in the DHCP response that points 169.254.269.254 to
> the VR IP?
> 
> Sent from my iPhone
> 
>> On Feb 25, 2017, at 3:18 AM, Nux! <nu...@li.nux.ro> wrote:
>> 
>> Hello,
>> 
>> The upstream Ubuntu and CentOS images will not work out of the box with
>> Cloudstack. I wanted to reuse them for my OpenVM project and failed.
>> 
>> The default metadata source is EC2 or Openstack, don't remember, cloud-init will
>> take ages to time-out and there is zero support for the root password feature -
>> in fact the cloud world is moving away from direct root access and so should we
>> - at least for compatibility reasons if not something else.
>> 
>> I am happy to put some more elbow grease into openvm.eu and make some
>> more/better templates.
>> 
>> I can also host other templates such as the cloud.com ones.
>> 
>> Open to comments, questions etc.
>> 
>> Lucian
>> 
>> --
>> Sent from the Delta quadrant using Borg technology!
>> 
>> Nux!
>> www.nux.ro
>> 
>> ----- Original Message -----
>>> From: "Chiradeep Vittal" <ch...@gmail.com>
>>> To: "dev" <de...@cloudstack.apache.org>
>>> Sent: Saturday, 25 February, 2017 00:18:48
>>> Subject: Re: Modern template hosting
>> 
>>> Also CentOS KVM images can be obtained here:
>>> http://cloud.centos.org/centos/7/images/
>>> 
>>> If these work, then the only question is where to host the systemvm images.
>>> 
>>> On Fri, Feb 24, 2017 at 4:05 PM, Chiradeep Vittal <ch...@gmail.com>
>>> wrote:
>>> 
>>>> I noticed that Ubuntu provides daily builds, including VHDs. Perhaps we
>>>> could point the default templates to their URL? Anyone willing to quickly
>>>> try out their images to see if they work on ACS?
>>>> 
>>>> https://cloud-images.ubuntu.com/xenial/current/
>>>> 
>>>> On Fri, Feb 24, 2017 at 9:53 AM, Will Stevens <ws...@cloudops.com>
>>>> wrote:
>>>> 
>>>>> unfortunately the template mirror conversation got caught up in details
>>>>> and
>>>>> nobody took the lead on implementing a solution.
>>>>> 
>>>>> citrix has been pinging me every couple months to say 'dude, we need to
>>>>> remove the dependency on download.citrix.com', but i have not had the
>>>>> cycles to get in and solve the problem.  the shutdown of that is imminent
>>>>> right now, so we need to solve it asap.
>>>>> 
>>>>> 
>>>>> 
>>>>> *Will STEVENS*
>>>>> Lead Developer
>>>>> 
>>>>> <https://goo.gl/NYZ8KK>
>>>>> 
>>>>> On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <pa...@shapeblue.com>
>>>>> wrote:
>>>>> 
>>>>>> Hi Nathan,
>>>>>> 
>>>>>> Ideally, if you put the template location in (or use a template defined
>>>>>> in)  test_data.py then the actual location can be overridden by anyone
>>>>>> testing.
>>>>>> 
>>>>>> For Trillian, we've copied all of the templates that people have define
>>>>> to
>>>>>> a local repo and then replace the URLs in test_data.py to reduce
>>>>> bandwidth
>>>>>> use and download times.
>>>>>> 
>>>>>> Ie:
>>>>>> 
>>>>>>            "bootableIso":
>>>>>>                {
>>>>>>                    "displaytext": "Test Bootable ISO",
>>>>>>                    "name": "testISO",
>>>>>>                    "bootable": True,
>>>>>>                    "ispublic": False,
>>>>>>                    "url": "{{ marvin_images_location
>>>>>> }}/TinyCore-current.iso",
>>>>>>                    "ostype": 'Other Linux (64-bit)',
>>>>>>                    "mode": 'HTTP_DOWNLOAD'
>>>>>>        },
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> I thought that we had come up with a solution for download.cloud.com,
>>>>> by
>>>>>> having a mirrorlist hosted in Community Apache 'space' with anyone able
>>>>> to
>>>>>> out themselves forward as a mirror.
>>>>>> But I must admit I lost track of whether anyone made the requisite
>>>>> changes
>>>>>> in code....
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Kind regards,
>>>>>> 
>>>>>> Paul Angus
>>>>>> 
>>>>>> paul.angus@shapeblue.com
>>>>>> www.shapeblue.com
>>>>>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>>>>> @shapeblue
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
>>>>>> Behalf Of Will Stevens
>>>>>> Sent: 24 February 2017 16:30
>>>>>> To: dev@cloudstack.apache.org
>>>>>> Subject: Re: Modern template hosting
>>>>>> 
>>>>>> this is a hard questions.  in general, we should be setting up a mirror
>>>>> on
>>>>>> some cloudstack/apache domain and then mirror to other provided
>>>>> templates.
>>>>>> 
>>>>>> we MUST come up with a solution to deprecate 'download.cloud.com', that
>>>>>> is going to be going away any day now.
>>>>>> 
>>>>>> i don't know the right way to solve this to be honest, but if you have
>>>>>> ideas, i am willing to help.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> *Will STEVENS*
>>>>>> Lead Developer
>>>>>> 
>>>>>> <https://goo.gl/NYZ8KK>
>>>>>> 
>>>>>> On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <nj...@ena.com>
>>>>> wrote:
>>>>>> 
>>>>>>> So not to re-open a can of worms, but I’m in a situation where I need
>>>>>>> to come up with a Marvin component test that depends on a template
>>>>>>> based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
>>>>>> Ubuntu 10.04 .
>>>>>>> I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
>>>>>>> from looking at the thread "Migrating CloudStack content from
>>>>>>> download.cloud.com” it looks like there is resistance to using this
>>>>> at
>>>>>>> least for hosting system vm templates over concerns of neutrality.
>>>>>>> Would this be suitable for a component test?  If not, what is a
>>>>> “blessed”
>>>>>>> template location?
>>>>>>> 
>>>>>>> Thanks in advance!
>>>>>>> 
>>>>>>> Nathan Johnson
>>>>>>> R&D Engineer
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 618 Grassmere Park Drive, Suite 12
>>>>>>> Nashville, TN 37211
>>>>>>> General Office: 615-312-6000
>>>>>>> 
>>>>>>> website | blog | support
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>

Re: Modern template hosting

Posted by Chiradeep Vittal <ch...@gmail.com>.

Is the issue that ACS doesn't support 169.254.169.254 as the metadata URL?

For those hypervisors that use the Linux bridge and iptables, it should be easy to add a DNAT rule to translate it to the VR IP? I'm talking about code change here. 
Or send down a static route in the DHCP response that points 169.254.269.254 to the VR IP? 

Sent from my iPhone

> On Feb 25, 2017, at 3:18 AM, Nux! <nu...@li.nux.ro> wrote:
> 
> Hello,
> 
> The upstream Ubuntu and CentOS images will not work out of the box with Cloudstack. I wanted to reuse them for my OpenVM project and failed.
> 
> The default metadata source is EC2 or Openstack, don't remember, cloud-init will take ages to time-out and there is zero support for the root password feature - in fact the cloud world is moving away from direct root access and so should we - at least for compatibility reasons if not something else.
> 
> I am happy to put some more elbow grease into openvm.eu and make some more/better templates. 
> 
> I can also host other templates such as the cloud.com ones.
> 
> Open to comments, questions etc.
> 
> Lucian
> 
> --
> Sent from the Delta quadrant using Borg technology!
> 
> Nux!
> www.nux.ro
> 
> ----- Original Message -----
>> From: "Chiradeep Vittal" <ch...@gmail.com>
>> To: "dev" <de...@cloudstack.apache.org>
>> Sent: Saturday, 25 February, 2017 00:18:48
>> Subject: Re: Modern template hosting
> 
>> Also CentOS KVM images can be obtained here:
>> http://cloud.centos.org/centos/7/images/
>> 
>> If these work, then the only question is where to host the systemvm images.
>> 
>> On Fri, Feb 24, 2017 at 4:05 PM, Chiradeep Vittal <ch...@gmail.com>
>> wrote:
>> 
>>> I noticed that Ubuntu provides daily builds, including VHDs. Perhaps we
>>> could point the default templates to their URL? Anyone willing to quickly
>>> try out their images to see if they work on ACS?
>>> 
>>> https://cloud-images.ubuntu.com/xenial/current/
>>> 
>>> On Fri, Feb 24, 2017 at 9:53 AM, Will Stevens <ws...@cloudops.com>
>>> wrote:
>>> 
>>>> unfortunately the template mirror conversation got caught up in details
>>>> and
>>>> nobody took the lead on implementing a solution.
>>>> 
>>>> citrix has been pinging me every couple months to say 'dude, we need to
>>>> remove the dependency on download.citrix.com', but i have not had the
>>>> cycles to get in and solve the problem.  the shutdown of that is imminent
>>>> right now, so we need to solve it asap.
>>>> 
>>>> 
>>>> 
>>>> *Will STEVENS*
>>>> Lead Developer
>>>> 
>>>> <https://goo.gl/NYZ8KK>
>>>> 
>>>> On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <pa...@shapeblue.com>
>>>> wrote:
>>>> 
>>>>> Hi Nathan,
>>>>> 
>>>>> Ideally, if you put the template location in (or use a template defined
>>>>> in)  test_data.py then the actual location can be overridden by anyone
>>>>> testing.
>>>>> 
>>>>> For Trillian, we've copied all of the templates that people have define
>>>> to
>>>>> a local repo and then replace the URLs in test_data.py to reduce
>>>> bandwidth
>>>>> use and download times.
>>>>> 
>>>>> Ie:
>>>>> 
>>>>>            "bootableIso":
>>>>>                {
>>>>>                    "displaytext": "Test Bootable ISO",
>>>>>                    "name": "testISO",
>>>>>                    "bootable": True,
>>>>>                    "ispublic": False,
>>>>>                    "url": "{{ marvin_images_location
>>>>> }}/TinyCore-current.iso",
>>>>>                    "ostype": 'Other Linux (64-bit)',
>>>>>                    "mode": 'HTTP_DOWNLOAD'
>>>>>        },
>>>>> 
>>>>> 
>>>>> 
>>>>> I thought that we had come up with a solution for download.cloud.com,
>>>> by
>>>>> having a mirrorlist hosted in Community Apache 'space' with anyone able
>>>> to
>>>>> out themselves forward as a mirror.
>>>>> But I must admit I lost track of whether anyone made the requisite
>>>> changes
>>>>> in code....
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> Kind regards,
>>>>> 
>>>>> Paul Angus
>>>>> 
>>>>> paul.angus@shapeblue.com
>>>>> www.shapeblue.com
>>>>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>>>> @shapeblue
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -----Original Message-----
>>>>> From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
>>>>> Behalf Of Will Stevens
>>>>> Sent: 24 February 2017 16:30
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: Re: Modern template hosting
>>>>> 
>>>>> this is a hard questions.  in general, we should be setting up a mirror
>>>> on
>>>>> some cloudstack/apache domain and then mirror to other provided
>>>> templates.
>>>>> 
>>>>> we MUST come up with a solution to deprecate 'download.cloud.com', that
>>>>> is going to be going away any day now.
>>>>> 
>>>>> i don't know the right way to solve this to be honest, but if you have
>>>>> ideas, i am willing to help.
>>>>> 
>>>>> 
>>>>> 
>>>>> *Will STEVENS*
>>>>> Lead Developer
>>>>> 
>>>>> <https://goo.gl/NYZ8KK>
>>>>> 
>>>>> On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <nj...@ena.com>
>>>> wrote:
>>>>> 
>>>>>> So not to re-open a can of worms, but I’m in a situation where I need
>>>>>> to come up with a Marvin component test that depends on a template
>>>>>> based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
>>>>> Ubuntu 10.04 .
>>>>>> I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
>>>>>> from looking at the thread "Migrating CloudStack content from
>>>>>> download.cloud.com” it looks like there is resistance to using this
>>>> at
>>>>>> least for hosting system vm templates over concerns of neutrality.
>>>>>> Would this be suitable for a component test?  If not, what is a
>>>> “blessed”
>>>>>> template location?
>>>>>> 
>>>>>> Thanks in advance!
>>>>>> 
>>>>>> Nathan Johnson
>>>>>> R&D Engineer
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 618 Grassmere Park Drive, Suite 12
>>>>>> Nashville, TN 37211
>>>>>> General Office: 615-312-6000
>>>>>> 
>>>>>> website | blog | support
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>

Re: Modern template hosting

Posted by Nux! <nu...@li.nux.ro>.

Hello,

The upstream Ubuntu and CentOS images will not work out of the box with Cloudstack. I wanted to reuse them for my OpenVM project and failed.

The default metadata source is EC2 or Openstack, don't remember, cloud-init will take ages to time-out and there is zero support for the root password feature - in fact the cloud world is moving away from direct root access and so should we - at least for compatibility reasons if not something else.

I am happy to put some more elbow grease into openvm.eu and make some more/better templates. 

I can also host other templates such as the cloud.com ones.

Open to comments, questions etc.

Lucian

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Chiradeep Vittal" <ch...@gmail.com>
> To: "dev" <de...@cloudstack.apache.org>
> Sent: Saturday, 25 February, 2017 00:18:48
> Subject: Re: Modern template hosting

> Also CentOS KVM images can be obtained here:
> http://cloud.centos.org/centos/7/images/
> 
> If these work, then the only question is where to host the systemvm images.
> 
> On Fri, Feb 24, 2017 at 4:05 PM, Chiradeep Vittal <ch...@gmail.com>
> wrote:
> 
>> I noticed that Ubuntu provides daily builds, including VHDs. Perhaps we
>> could point the default templates to their URL? Anyone willing to quickly
>> try out their images to see if they work on ACS?
>>
>> https://cloud-images.ubuntu.com/xenial/current/
>>
>> On Fri, Feb 24, 2017 at 9:53 AM, Will Stevens <ws...@cloudops.com>
>> wrote:
>>
>>> unfortunately the template mirror conversation got caught up in details
>>> and
>>> nobody took the lead on implementing a solution.
>>>
>>> citrix has been pinging me every couple months to say 'dude, we need to
>>> remove the dependency on download.citrix.com', but i have not had the
>>> cycles to get in and solve the problem.  the shutdown of that is imminent
>>> right now, so we need to solve it asap.
>>>
>>>
>>>
>>> *Will STEVENS*
>>> Lead Developer
>>>
>>> <https://goo.gl/NYZ8KK>
>>>
>>> On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <pa...@shapeblue.com>
>>> wrote:
>>>
>>> > Hi Nathan,
>>> >
>>> > Ideally, if you put the template location in (or use a template defined
>>> > in)  test_data.py then the actual location can be overridden by anyone
>>> > testing.
>>> >
>>> > For Trillian, we've copied all of the templates that people have define
>>> to
>>> > a local repo and then replace the URLs in test_data.py to reduce
>>> bandwidth
>>> > use and download times.
>>> >
>>> > Ie:
>>> >
>>> >             "bootableIso":
>>> >                 {
>>> >                     "displaytext": "Test Bootable ISO",
>>> >                     "name": "testISO",
>>> >                     "bootable": True,
>>> >                     "ispublic": False,
>>> >                     "url": "{{ marvin_images_location
>>> > }}/TinyCore-current.iso",
>>> >                     "ostype": 'Other Linux (64-bit)',
>>> >                     "mode": 'HTTP_DOWNLOAD'
>>> >         },
>>> >
>>> >
>>> >
>>> > I thought that we had come up with a solution for download.cloud.com,
>>> by
>>> > having a mirrorlist hosted in Community Apache 'space' with anyone able
>>> to
>>> > out themselves forward as a mirror.
>>> > But I must admit I lost track of whether anyone made the requisite
>>> changes
>>> > in code....
>>> >
>>> >
>>> >
>>> >
>>> > Kind regards,
>>> >
>>> > Paul Angus
>>> >
>>> > paul.angus@shapeblue.com
>>> > www.shapeblue.com
>>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>> > @shapeblue
>>> >
>>> >
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
>>> > Behalf Of Will Stevens
>>> > Sent: 24 February 2017 16:30
>>> > To: dev@cloudstack.apache.org
>>> > Subject: Re: Modern template hosting
>>> >
>>> > this is a hard questions.  in general, we should be setting up a mirror
>>> on
>>> > some cloudstack/apache domain and then mirror to other provided
>>> templates.
>>> >
>>> > we MUST come up with a solution to deprecate 'download.cloud.com', that
>>> > is going to be going away any day now.
>>> >
>>> > i don't know the right way to solve this to be honest, but if you have
>>> > ideas, i am willing to help.
>>> >
>>> >
>>> >
>>> > *Will STEVENS*
>>> > Lead Developer
>>> >
>>> > <https://goo.gl/NYZ8KK>
>>> >
>>> > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <nj...@ena.com>
>>> wrote:
>>> >
>>> > > So not to re-open a can of worms, but I’m in a situation where I need
>>> > > to come up with a Marvin component test that depends on a template
>>> > > based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
>>> > Ubuntu 10.04 .
>>> > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
>>> > > from looking at the thread "Migrating CloudStack content from
>>> > > download.cloud.com” it looks like there is resistance to using this
>>> at
>>> > > least for hosting system vm templates over concerns of neutrality.
>>> > > Would this be suitable for a component test?  If not, what is a
>>> “blessed”
>>> > > template location?
>>> > >
>>> > > Thanks in advance!
>>> > >
>>> > > Nathan Johnson
>>> > > R&D Engineer
>>> > >
>>> > >
>>> > >
>>> > > 618 Grassmere Park Drive, Suite 12
>>> > > Nashville, TN 37211
>>> > > General Office: 615-312-6000
>>> > >
>>> > > website | blog | support
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> >
>>>
>>

Re: Modern template hosting

Posted by Chiradeep Vittal <ch...@gmail.com>.

Also CentOS KVM images can be obtained here:
http://cloud.centos.org/centos/7/images/

If these work, then the only question is where to host the systemvm images.

On Fri, Feb 24, 2017 at 4:05 PM, Chiradeep Vittal <ch...@gmail.com>
wrote:

> I noticed that Ubuntu provides daily builds, including VHDs. Perhaps we
> could point the default templates to their URL? Anyone willing to quickly
> try out their images to see if they work on ACS?
>
> https://cloud-images.ubuntu.com/xenial/current/
>
> On Fri, Feb 24, 2017 at 9:53 AM, Will Stevens <ws...@cloudops.com>
> wrote:
>
>> unfortunately the template mirror conversation got caught up in details
>> and
>> nobody took the lead on implementing a solution.
>>
>> citrix has been pinging me every couple months to say 'dude, we need to
>> remove the dependency on download.citrix.com', but i have not had the
>> cycles to get in and solve the problem.  the shutdown of that is imminent
>> right now, so we need to solve it asap.
>>
>>
>>
>> *Will STEVENS*
>> Lead Developer
>>
>> <https://goo.gl/NYZ8KK>
>>
>> On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <pa...@shapeblue.com>
>> wrote:
>>
>> > Hi Nathan,
>> >
>> > Ideally, if you put the template location in (or use a template defined
>> > in)  test_data.py then the actual location can be overridden by anyone
>> > testing.
>> >
>> > For Trillian, we've copied all of the templates that people have define
>> to
>> > a local repo and then replace the URLs in test_data.py to reduce
>> bandwidth
>> > use and download times.
>> >
>> > Ie:
>> >
>> >             "bootableIso":
>> >                 {
>> >                     "displaytext": "Test Bootable ISO",
>> >                     "name": "testISO",
>> >                     "bootable": True,
>> >                     "ispublic": False,
>> >                     "url": "{{ marvin_images_location
>> > }}/TinyCore-current.iso",
>> >                     "ostype": 'Other Linux (64-bit)',
>> >                     "mode": 'HTTP_DOWNLOAD'
>> >         },
>> >
>> >
>> >
>> > I thought that we had come up with a solution for download.cloud.com,
>> by
>> > having a mirrorlist hosted in Community Apache 'space' with anyone able
>> to
>> > out themselves forward as a mirror.
>> > But I must admit I lost track of whether anyone made the requisite
>> changes
>> > in code....
>> >
>> >
>> >
>> >
>> > Kind regards,
>> >
>> > Paul Angus
>> >
>> > paul.angus@shapeblue.com
>> > www.shapeblue.com
>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > @shapeblue
>> >
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
>> > Behalf Of Will Stevens
>> > Sent: 24 February 2017 16:30
>> > To: dev@cloudstack.apache.org
>> > Subject: Re: Modern template hosting
>> >
>> > this is a hard questions.  in general, we should be setting up a mirror
>> on
>> > some cloudstack/apache domain and then mirror to other provided
>> templates.
>> >
>> > we MUST come up with a solution to deprecate 'download.cloud.com', that
>> > is going to be going away any day now.
>> >
>> > i don't know the right way to solve this to be honest, but if you have
>> > ideas, i am willing to help.
>> >
>> >
>> >
>> > *Will STEVENS*
>> > Lead Developer
>> >
>> > <https://goo.gl/NYZ8KK>
>> >
>> > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <nj...@ena.com>
>> wrote:
>> >
>> > > So not to re-open a can of worms, but I’m in a situation where I need
>> > > to come up with a Marvin component test that depends on a template
>> > > based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
>> > Ubuntu 10.04 .
>> > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
>> > > from looking at the thread "Migrating CloudStack content from
>> > > download.cloud.com” it looks like there is resistance to using this
>> at
>> > > least for hosting system vm templates over concerns of neutrality.
>> > > Would this be suitable for a component test?  If not, what is a
>> “blessed”
>> > > template location?
>> > >
>> > > Thanks in advance!
>> > >
>> > > Nathan Johnson
>> > > R&D Engineer
>> > >
>> > >
>> > >
>> > > 618 Grassmere Park Drive, Suite 12
>> > > Nashville, TN 37211
>> > > General Office: 615-312-6000
>> > >
>> > > website | blog | support
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> >
>>
>
>

Re: Modern template hosting

Posted by Chiradeep Vittal <ch...@gmail.com>.

I noticed that Ubuntu provides daily builds, including VHDs. Perhaps we
could point the default templates to their URL? Anyone willing to quickly
try out their images to see if they work on ACS?

https://cloud-images.ubuntu.com/xenial/current/

On Fri, Feb 24, 2017 at 9:53 AM, Will Stevens <ws...@cloudops.com> wrote:

> unfortunately the template mirror conversation got caught up in details and
> nobody took the lead on implementing a solution.
>
> citrix has been pinging me every couple months to say 'dude, we need to
> remove the dependency on download.citrix.com', but i have not had the
> cycles to get in and solve the problem.  the shutdown of that is imminent
> right now, so we need to solve it asap.
>
>
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <pa...@shapeblue.com>
> wrote:
>
> > Hi Nathan,
> >
> > Ideally, if you put the template location in (or use a template defined
> > in)  test_data.py then the actual location can be overridden by anyone
> > testing.
> >
> > For Trillian, we've copied all of the templates that people have define
> to
> > a local repo and then replace the URLs in test_data.py to reduce
> bandwidth
> > use and download times.
> >
> > Ie:
> >
> >             "bootableIso":
> >                 {
> >                     "displaytext": "Test Bootable ISO",
> >                     "name": "testISO",
> >                     "bootable": True,
> >                     "ispublic": False,
> >                     "url": "{{ marvin_images_location
> > }}/TinyCore-current.iso",
> >                     "ostype": 'Other Linux (64-bit)',
> >                     "mode": 'HTTP_DOWNLOAD'
> >         },
> >
> >
> >
> > I thought that we had come up with a solution for download.cloud.com, by
> > having a mirrorlist hosted in Community Apache 'space' with anyone able
> to
> > out themselves forward as a mirror.
> > But I must admit I lost track of whether anyone made the requisite
> changes
> > in code....
> >
> >
> >
> >
> > Kind regards,
> >
> > Paul Angus
> >
> > paul.angus@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> > -----Original Message-----
> > From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
> > Behalf Of Will Stevens
> > Sent: 24 February 2017 16:30
> > To: dev@cloudstack.apache.org
> > Subject: Re: Modern template hosting
> >
> > this is a hard questions.  in general, we should be setting up a mirror
> on
> > some cloudstack/apache domain and then mirror to other provided
> templates.
> >
> > we MUST come up with a solution to deprecate 'download.cloud.com', that
> > is going to be going away any day now.
> >
> > i don't know the right way to solve this to be honest, but if you have
> > ideas, i am willing to help.
> >
> >
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <nj...@ena.com>
> wrote:
> >
> > > So not to re-open a can of worms, but I’m in a situation where I need
> > > to come up with a Marvin component test that depends on a template
> > > based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
> > Ubuntu 10.04 .
> > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
> > > from looking at the thread "Migrating CloudStack content from
> > > download.cloud.com” it looks like there is resistance to using this at
> > > least for hosting system vm templates over concerns of neutrality.
> > > Would this be suitable for a component test?  If not, what is a
> “blessed”
> > > template location?
> > >
> > > Thanks in advance!
> > >
> > > Nathan Johnson
> > > R&D Engineer
> > >
> > >
> > >
> > > 618 Grassmere Park Drive, Suite 12
> > > Nashville, TN 37211
> > > General Office: 615-312-6000
> > >
> > > website | blog | support
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
>

Re: Modern template hosting

Posted by Will Stevens <ws...@cloudops.com>.

unfortunately the template mirror conversation got caught up in details and
nobody took the lead on implementing a solution.

citrix has been pinging me every couple months to say 'dude, we need to
remove the dependency on download.citrix.com', but i have not had the
cycles to get in and solve the problem.  the shutdown of that is imminent
right now, so we need to solve it asap.



*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <pa...@shapeblue.com>
wrote:

> Hi Nathan,
>
> Ideally, if you put the template location in (or use a template defined
> in)  test_data.py then the actual location can be overridden by anyone
> testing.
>
> For Trillian, we've copied all of the templates that people have define to
> a local repo and then replace the URLs in test_data.py to reduce bandwidth
> use and download times.
>
> Ie:
>
>             "bootableIso":
>                 {
>                     "displaytext": "Test Bootable ISO",
>                     "name": "testISO",
>                     "bootable": True,
>                     "ispublic": False,
>                     "url": "{{ marvin_images_location
> }}/TinyCore-current.iso",
>                     "ostype": 'Other Linux (64-bit)',
>                     "mode": 'HTTP_DOWNLOAD'
>         },
>
>
>
> I thought that we had come up with a solution for download.cloud.com, by
> having a mirrorlist hosted in Community Apache 'space' with anyone able to
> out themselves forward as a mirror.
> But I must admit I lost track of whether anyone made the requisite changes
> in code....
>
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
> Behalf Of Will Stevens
> Sent: 24 February 2017 16:30
> To: dev@cloudstack.apache.org
> Subject: Re: Modern template hosting
>
> this is a hard questions.  in general, we should be setting up a mirror on
> some cloudstack/apache domain and then mirror to other provided templates.
>
> we MUST come up with a solution to deprecate 'download.cloud.com', that
> is going to be going away any day now.
>
> i don't know the right way to solve this to be honest, but if you have
> ideas, i am willing to help.
>
>
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <nj...@ena.com> wrote:
>
> > So not to re-open a can of worms, but I’m in a situation where I need
> > to come up with a Marvin component test that depends on a template
> > based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
> Ubuntu 10.04 .
> > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
> > from looking at the thread "Migrating CloudStack content from
> > download.cloud.com” it looks like there is resistance to using this at
> > least for hosting system vm templates over concerns of neutrality.
> > Would this be suitable for a component test?  If not, what is a “blessed”
> > template location?
> >
> > Thanks in advance!
> >
> > Nathan Johnson
> > R&D Engineer
> >
> >
> >
> > 618 Grassmere Park Drive, Suite 12
> > Nashville, TN 37211
> > General Office: 615-312-6000
> >
> > website | blog | support
> >
> >
> >
> >
> >
> >
> >
>

RE: Modern template hosting

Posted by Paul Angus <pa...@shapeblue.com>.

Hi Nathan,

Ideally, if you put the template location in (or use a template defined in)  test_data.py then the actual location can be overridden by anyone testing. 

For Trillian, we've copied all of the templates that people have define to a local repo and then replace the URLs in test_data.py to reduce bandwidth use and download times.

Ie:

            "bootableIso":
                {
                    "displaytext": "Test Bootable ISO",
                    "name": "testISO",
                    "bootable": True,
                    "ispublic": False,
                    "url": "{{ marvin_images_location }}/TinyCore-current.iso",
                    "ostype": 'Other Linux (64-bit)',
                    "mode": 'HTTP_DOWNLOAD'
        },

I thought that we had come up with a solution for download.cloud.com, by having a mirrorlist hosted in Community Apache 'space' with anyone able to out themselves forward as a mirror.
But I must admit I lost track of whether anyone made the requisite changes in code....

Kind regards,

Paul Angus

paul.angus@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue

-----Original Message-----
From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On Behalf Of Will Stevens
Sent: 24 February 2017 16:30
To: dev@cloudstack.apache.org
Subject: Re: Modern template hosting

this is a hard questions.  in general, we should be setting up a mirror on some cloudstack/apache domain and then mirror to other provided templates.

we MUST come up with a solution to deprecate 'download.cloud.com', that is going to be going away any day now.

i don't know the right way to solve this to be honest, but if you have ideas, i am willing to help.

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <nj...@ena.com> wrote:

> So not to re-open a can of worms, but I’m in a situation where I need 
> to come up with a Marvin component test that depends on a template 
> based on a kernel that’s relatively new, i.e., newer than Centos 5.3 / Ubuntu 10.04 .
> I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but 
> from looking at the thread "Migrating CloudStack content from 
> download.cloud.com” it looks like there is resistance to using this at 
> least for hosting system vm templates over concerns of neutrality.  
> Would this be suitable for a component test?  If not, what is a “blessed”
> template location?
>
> Thanks in advance!
>
> Nathan Johnson
> R&D Engineer
>
>
>
> 618 Grassmere Park Drive, Suite 12
> Nashville, TN 37211
> General Office: 615-312-6000
>
> website | blog | support
>
>
>
>
>
>
>

Re: Modern template hosting

Posted by Will Stevens <ws...@cloudops.com>.

this is a hard questions.  in general, we should be setting up a mirror on
some cloudstack/apache domain and then mirror to other provided templates.

we MUST come up with a solution to deprecate 'download.cloud.com', that is
going to be going away any day now.

i don't know the right way to solve this to be honest, but if you have
ideas, i am willing to help.

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <nj...@ena.com> wrote:

> So not to re-open a can of worms, but I’m in a situation where I need to
> come up with a Marvin component test that depends on a template based on a
> kernel that’s relatively new, i.e., newer than Centos 5.3 / Ubuntu 10.04 .
> I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but from
> looking at the thread "Migrating CloudStack content from
> download.cloud.com” it looks like there is resistance to using this at
> least for hosting system vm templates over concerns of neutrality.  Would
> this be suitable for a component test?  If not, what is a “blessed”
> template location?
>
> Thanks in advance!
>
> Nathan Johnson
> R&D Engineer
>
>
>
> 618 Grassmere Park Drive, Suite 12
> Nashville, TN 37211
> General Office: 615-312-6000
>
> website | blog | support
>
>
>
>
>
>
>