You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openoffice.apache.org by Rob Weir <ro...@apache.org> on 2011/11/29 17:06:39 UTC
Extensions and templates
On Tue, Nov 29, 2011 at 8:51 AM, Gavin McDonald <ga...@16degrees.com.au> wrote:
>
>
>> -----Original Message-----
>> From: Rob Weir [mailto:robweir@apache.org]
>> Sent: Tuesday, 29 November 2011 10:15 PM
>> To: ooo-dev@incubator.apache.org
>> Subject: Re: Business models that will not work [was: Re: Can we update our
>> migration status table?]
>>
>> On Mon, Nov 28, 2011 at 10:09 PM, Pedro Giffuni <pf...@apache.org> wrote:
>> > FWIW;
>> >
>> > I think a central site for hosting AOO extensions would be welcome. It
>> > would be fine to have such a site sponsored by donations and web
>> > publicity, and offering a share of technical support and commercial
>> > extensions would be fine too.
>> >
>>
>> Compare, for example, Apache Maven with Sonatype's Maven Central.
>> This is similar to AOO and Extensions.
>
> I haven’t really read this thread, but I have a server here in AU if there really is
> nowhere else to host this stuff. Are there any specs on what's required?
>
> Or, shoot me if I got the wrong end of the stick.
>
Yes, that is the wrong end of the stick. But it is an interesting
side of the stick as well, so I'll move it to its own thread, so we
can explore the options.
We currently have an extensions repository and a template repository,
kindly hosted by OSUOSL (http://osuosl.org/) on our subdomains:
extensions.services.openoffice.org
templates.services.openoffice.org
These repositories carry 3rd extensions and templates under a wide
variety of licenses, including a mix of open source licenses, copyleft
and non-copyleft, as well as non-open source licensed "freeware" and
"trial-ware" packages. Because of these various licenses, I think it
is unlikely to be something that we could host at Apache, even in just
binary form.
So what are our options?
==Option 1: Remain at OSUOSL==
We could remain with OSUOSL hosting. However, the existing site is
very unstable. For this approach to be practical we'd need a
volunteer with Drupal skills to work with OSUOSL to diagnose what is
wrong and to restore stability to these services. Maybe some sight
maintenance, upgrades or tuning is sufficient?
I think this would be the ideal short-term solution at the very least.
==Option 2: Move Critical extensions to stable host==
This is more a back up plan if nothing else works in the AOO 3.4
timeframe. There are a handful of critical extensions that many users
will want access to, such as spell checking dictionaries. If OSUOSL
is not stable when 3.4 is released, then we will have many thousands
of very frustrated users. So with this option we copy the critical
extensions to Apache-Extras and point 3.4 users to that.
==Option 3: Clone OSUOSL repositories to another host==
In this option we rehost the existing repositories at another host.
So similar to Option 1, we would need a volunteer with Drupal
expertise.
==Option 4: Host repositories elsewhere, using new UI==
SourceForge, etc. There are ways, for example, to create
meta-communities ("Neighborhoods") at SourceForge across projects.
Similar things can be done at Google Code or elsewhere. Take
advantage of these forges rather than maintaining our own customized
app.
==Option 5: Re-architect the Repositories==
This is the option I personally favor for long term. The downside of
what we have today is we have a single repository with a single host.
This is non-optimal for several reasons. There are different needs
out there if we look at downstream consumers and/or enterprise users.
They may want to have a restricted or supplemental repository. I may
want to make available to my users an interface that shows only
non-proprietary extensions, plus non copyleft templates plus my own
proprietary house templates. Someone else might want to restrict
things differently. Look at analogous things with Ubuntu component
repositories, with the ability to disable or enable proprietary
drivers and/or non-supported components.
In a new design we could start up front with a specification for a
data model for an extension, something expressed in simple XML, with a
query/fetch interface as a RESTful service. This would allow multiple
repositories to look and behave identically from the data perspective.
That then allows websites that aggregate such repository data, as
well as in-app browsers for extensions and templates. Then you can
imagine AOO allowing a user or admin to enable any from a list of
several extension repositories to work with.
The other thing this approach does is separate the extension metadata
from the actual licensed extension. If we wanted to have a canonical
repository of registered extensions, but without actually hosting or
storing the extensions, then that should be OK. We're hosting URL's
to resources. We're not distributing code.
(There might be some hybrid way of doing this as well, by using
SourceForge or similar for the underlying hosting, but then with tags
or a common extension.xml in the root of each repository, then
publishing data that we create our catalog from)
> (Happy to do the migration too if that’s needed)
>
> Gav...
RE: Extensions and templates
Posted by Gavin McDonald <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Rob Weir [mailto:robweir@apache.org]
> Sent: Wednesday, 30 November 2011 4:02 AM
> To: ooo-dev@incubator.apache.org
> Subject: Re: Extensions and templates
>
> On Tue, Nov 29, 2011 at 12:51 PM, Dave Fisher <da...@comcast.net>
> wrote:
> >
> > On Nov 29, 2011, at 9:24 AM, Joost Andrae wrote:
> >
> >> Hi Rob,
> >>
> >>
> >>> So what are our options?
> >>>
> >>> ==Option 1: Remain at OSUOSL==
> >>>
> >>> We could remain with OSUOSL hosting. However, the existing site is
> >>> very unstable. For this approach to be practical we'd need a
> >>> volunteer with Drupal skills to work with OSUOSL to diagnose what is
> >>> wrong and to restore stability to these services. Maybe some sight
> >>> maintenance, upgrades or tuning is sufficient?
> >>>
> >>> I think this would be the ideal short-term solution at the very least.
> >>
> >> I think option one seems to be the most convenient way.
> >> Maybe someone could talk to Lance Albertson at OSUOSL (contact via
> >> support@osuosl.org or via IRC; better look at
> >> http://osuosl.org/contact)
> >
> > +1, Lance has been very supportive. He can explain all about varnish
> locking, nagios being turned off, and the two different versions of Drupal on
> each server.
> >
>
> Well.... that has been the obvious option for several months now, but no one
> has stepped up. So maybe it is not as obvious as we think it is? I'm laying out
> the other options in case someone finds another option more interesting.
> Sometimes an interesting problem to solve will attract volunteers where
> "cleanup in aisle 5" won't, thought it may be objectively the simpler
> approach.
>
Fine, I bite, let me talk to Lance and see what the score is and I'll get back to you.
Gav...
> -Rob
>
> > Best Regards,
> > Dave
> >
> >
> >>
> >> Kind regards, Joost
> >>
> >
Re: Extensions and templates
Posted by Rob Weir <ro...@apache.org>.
On Tue, Nov 29, 2011 at 12:51 PM, Dave Fisher <da...@comcast.net> wrote:
>
> On Nov 29, 2011, at 9:24 AM, Joost Andrae wrote:
>
>> Hi Rob,
>>
>>
>>> So what are our options?
>>>
>>> ==Option 1: Remain at OSUOSL==
>>>
>>> We could remain with OSUOSL hosting. However, the existing site is
>>> very unstable. For this approach to be practical we'd need a
>>> volunteer with Drupal skills to work with OSUOSL to diagnose what is
>>> wrong and to restore stability to these services. Maybe some sight
>>> maintenance, upgrades or tuning is sufficient?
>>>
>>> I think this would be the ideal short-term solution at the very least.
>>
>> I think option one seems to be the most convenient way.
>> Maybe someone could talk to Lance Albertson at OSUOSL (contact via support@osuosl.org or via IRC; better look at http://osuosl.org/contact)
>
> +1, Lance has been very supportive. He can explain all about varnish locking, nagios being turned off, and the two different versions of Drupal on each server.
>
Well.... that has been the obvious option for several months now, but
no one has stepped up. So maybe it is not as obvious as we think it
is? I'm laying out the other options in case someone finds another
option more interesting. Sometimes an interesting problem to solve
will attract volunteers where "cleanup in aisle 5" won't, thought it
may be objectively the simpler approach.
-Rob
> Best Regards,
> Dave
>
>
>>
>> Kind regards, Joost
>>
>
Re: Extensions and templates
Posted by Dave Fisher <da...@comcast.net>.
On Nov 29, 2011, at 9:24 AM, Joost Andrae wrote:
> Hi Rob,
>
>
>> So what are our options?
>>
>> ==Option 1: Remain at OSUOSL==
>>
>> We could remain with OSUOSL hosting. However, the existing site is
>> very unstable. For this approach to be practical we'd need a
>> volunteer with Drupal skills to work with OSUOSL to diagnose what is
>> wrong and to restore stability to these services. Maybe some sight
>> maintenance, upgrades or tuning is sufficient?
>>
>> I think this would be the ideal short-term solution at the very least.
>
> I think option one seems to be the most convenient way.
> Maybe someone could talk to Lance Albertson at OSUOSL (contact via support@osuosl.org or via IRC; better look at http://osuosl.org/contact)
+1, Lance has been very supportive. He can explain all about varnish locking, nagios being turned off, and the two different versions of Drupal on each server.
Best Regards,
Dave
>
> Kind regards, Joost
>
Re: Extensions and templates
Posted by Joost Andrae <Jo...@gmx.de>.
Hi Rob,
> So what are our options?
>
> ==Option 1: Remain at OSUOSL==
>
> We could remain with OSUOSL hosting. However, the existing site is
> very unstable. For this approach to be practical we'd need a
> volunteer with Drupal skills to work with OSUOSL to diagnose what is
> wrong and to restore stability to these services. Maybe some sight
> maintenance, upgrades or tuning is sufficient?
>
> I think this would be the ideal short-term solution at the very least.
I think option one seems to be the most convenient way.
Maybe someone could talk to Lance Albertson at OSUOSL (contact via
support@osuosl.org or via IRC; better look at http://osuosl.org/contact)
Kind regards, Joost
Re: Extensions and templates
Posted by Jürgen Schmidt <jo...@googlemail.com>.
On 11/29/11 5:06 PM, Rob Weir wrote:
> On Tue, Nov 29, 2011 at 8:51 AM, Gavin McDonald<ga...@16degrees.com.au> wrote:
>>
>>
>>> -----Original Message-----
>>> From: Rob Weir [mailto:robweir@apache.org]
>>> Sent: Tuesday, 29 November 2011 10:15 PM
>>> To: ooo-dev@incubator.apache.org
>>> Subject: Re: Business models that will not work [was: Re: Can we update our
>>> migration status table?]
>>>
>>> On Mon, Nov 28, 2011 at 10:09 PM, Pedro Giffuni<pf...@apache.org> wrote:
>>>> FWIW;
>>>>
>>>> I think a central site for hosting AOO extensions would be welcome. It
>>>> would be fine to have such a site sponsored by donations and web
>>>> publicity, and offering a share of technical support and commercial
>>>> extensions would be fine too.
>>>>
>>>
>>> Compare, for example, Apache Maven with Sonatype's Maven Central.
>>> This is similar to AOO and Extensions.
>>
>> I haven’t really read this thread, but I have a server here in AU if there really is
>> nowhere else to host this stuff. Are there any specs on what's required?
>>
>> Or, shoot me if I got the wrong end of the stick.
>>
>
> Yes, that is the wrong end of the stick. But it is an interesting
> side of the stick as well, so I'll move it to its own thread, so we
> can explore the options.
>
> We currently have an extensions repository and a template repository,
> kindly hosted by OSUOSL (http://osuosl.org/) on our subdomains:
>
> extensions.services.openoffice.org
>
> templates.services.openoffice.org
>
> These repositories carry 3rd extensions and templates under a wide
> variety of licenses, including a mix of open source licenses, copyleft
> and non-copyleft, as well as non-open source licensed "freeware" and
> "trial-ware" packages. Because of these various licenses, I think it
> is unlikely to be something that we could host at Apache, even in just
> binary form.
>
> So what are our options?
>
> ==Option 1: Remain at OSUOSL==
>
> We could remain with OSUOSL hosting. However, the existing site is
> very unstable. For this approach to be practical we'd need a
> volunteer with Drupal skills to work with OSUOSL to diagnose what is
> wrong and to restore stability to these services. Maybe some sight
> maintenance, upgrades or tuning is sufficient?
>
> I think this would be the ideal short-term solution at the very least.
>
> ==Option 2: Move Critical extensions to stable host==
>
> This is more a back up plan if nothing else works in the AOO 3.4
> timeframe. There are a handful of critical extensions that many users
> will want access to, such as spell checking dictionaries. If OSUOSL
> is not stable when 3.4 is released, then we will have many thousands
> of very frustrated users. So with this option we copy the critical
> extensions to Apache-Extras and point 3.4 users to that.
>
> ==Option 3: Clone OSUOSL repositories to another host==
>
> In this option we rehost the existing repositories at another host.
> So similar to Option 1, we would need a volunteer with Drupal
> expertise.
>
> ==Option 4: Host repositories elsewhere, using new UI==
>
> SourceForge, etc. There are ways, for example, to create
> meta-communities ("Neighborhoods") at SourceForge across projects.
> Similar things can be done at Google Code or elsewhere. Take
> advantage of these forges rather than maintaining our own customized
> app.
>
> ==Option 5: Re-architect the Repositories==
>
> This is the option I personally favor for long term. The downside of
> what we have today is we have a single repository with a single host.
> This is non-optimal for several reasons. There are different needs
> out there if we look at downstream consumers and/or enterprise users.
> They may want to have a restricted or supplemental repository. I may
> want to make available to my users an interface that shows only
> non-proprietary extensions, plus non copyleft templates plus my own
> proprietary house templates. Someone else might want to restrict
> things differently. Look at analogous things with Ubuntu component
> repositories, with the ability to disable or enable proprietary
> drivers and/or non-supported components.
>
> In a new design we could start up front with a specification for a
> data model for an extension, something expressed in simple XML, with a
> query/fetch interface as a RESTful service. This would allow multiple
> repositories to look and behave identically from the data perspective.
> That then allows websites that aggregate such repository data, as
> well as in-app browsers for extensions and templates. Then you can
> imagine AOO allowing a user or admin to enable any from a list of
> several extension repositories to work with.
>
> The other thing this approach does is separate the extension metadata
> from the actual licensed extension. If we wanted to have a canonical
> repository of registered extensions, but without actually hosting or
> storing the extensions, then that should be OK. We're hosting URL's
> to resources. We're not distributing code.
>
> (There might be some hybrid way of doing this as well, by using
> SourceForge or similar for the underlying hosting, but then with tags
> or a common extension.xml in the root of each repository, then
> publishing data that we create our catalog from)
>
today we have one default repository but each extensions can carry it's
own update Url. That means in detail
- no update Url in the extension package, the default Url is used to
search for update information for this extension
- update Url in the package, this one is used to search for update
information.
Such an update Url can link to a simple xml file providing the
extensions meta data about available updates. This file can of course
provide information about more than one extension.
Based on this we can probably improve the whole extension management a
lot. And with some extra work we can provide a filtering and configuring
mechanism that you probably have in mind.
We should keep in mind that we do also some virus check (as far as i
know) for the extensions hosted on the OSUOSL repo.
Independent which way we prefer in the future, but we should ensure that
we have some control over the submitted extensions. I don't know how
exactly this can be ensured. I think about something like a review
process, or certifying. Simply to ensure that at least in an official
repo we don't have any malware.
Juergen
>
>> (Happy to do the migration too if that’s needed)
>>
>> Gav...
RE: Extensions and templates
Posted by Gavin McDonald <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Andrea Pescetti [mailto:pescetti@openoffice.org]
> Sent: Thursday, 8 December 2011 10:01 PM
> To: ooo-dev@incubator.apache.org
> Subject: Re: Extensions and templates
>
> Gavin McDonald wrote:
> >> From: Andrea Pescetti
> >> On 29/11/2011 Rob Weir wrote:
> >>> ==Option 1: Remain at OSUOSL==
> >>> We could remain with OSUOSL hosting. However, the existing site is
> >>> very unstable.
> >> This would be best both for short and long term. ...
> > Sorry, OSUOSL don’t want anything to do with these any longer
>
> As I wrote, Options 1 and 3 (i.e., staying with OSUOSL or cloning to another
> host) are fundamentally equivalent to me. My point is that the best first step
> is starting from the current websites or a clone of them.
>
> > The hosts themselves cannot cope with all the memory and cpu these are
> > consuming all the time, let alone the bandwidth.
>
> Then someone should explain why they were absolutely stable in 2010, with
> a traffic that can be safely assumed to be similar to the one they are receiving
> in 2011. Something broke, and since the Drupal code hasn't changed I still
> think that the malfunctioning components are somewhere else (or, if in
> Drupal, not in the site itself but in the interface to caching engines).
No clue, I wasn’t involved at that time.
>
> > I've had a look around
> > the drupal sites and it is not optimal to say the least.
>
> It would be helpful to know what level of access you obtained to make this
> assessment. Could you read the site code, or did you receive administrator
> credentials for the website, or did you get shell access to the machine?
Shell access to the machine, it is the code/files I'm talking about, rather than
the usage /admin of the site itself (at this stage).
>
> That the sites are not optimal is fairly obvious, especially considering that
> Extensions is a Drupal 5 site and thus creates sessions even for anonymous
> users; Drupal 7 is much better in this respect and offers more scalability out
> of the box and better integration with caching engines, so it seems a natural
> candidate for medium-term and long-term improvements ("Option 5").
Agree.
>
> >>> ==Option 2: Move Critical extensions to stable host==
> >> Indeed, as you write, this would be an extreme option.
> > More extreme would be to do nothing, as you'll end up with nothing.
>
> Of course. What I meant to say was: cherry-picking "important"
> extensions and creating a repository for them from scratch is more or less
> the same work of Option 1 or 3 (i.e., fix the current site or a clone of it) for a
> much less interesting result.
>
> >>> ==Option 3: Clone OSUOSL repositories to another host==
> >> This is not significantly different from Option 1; i.e., if there are
> >> other hosting options available the mere cloning of the site would
> >> not take long, but again the problem is not with the site but with caching.
> > Do not blame caches for poor performance. the caches are improving a
> > bad situation
>
> OK, no matter what we think about the root cause for the current bad
> performance it seems that we both agree that cloning the site will give us the
> possibility to tweak it (or the environment) and improve performance. Since
> I've seen it working perfectly in 2010, I'm confident this is achievable.
cool
>
> >>> ==Option 4: Host repositories elsewhere, using new UI==
> >> As I used to say, everybody who thinks that the Extensions or
> >> Templates sites can be replaced easily has never tried submitting a
> template!
> >> Thorsten did a lot of customization work on the two sites; any
> >> replacement would provide a largely inferior user experience.
> > I think you don’t think very highly of other peoples abilities, a poor outlook.
>
> That was just a warning: people should know (and they would know, if they
> had ever uploaded a template...) that the sites extract and use a lot of
> information specific to OpenOffice.org and ODF files. This is often overlooked
> when people see these sites as "some form of file repositories" and propose
> to rely on different solutions: they should be aware that, to provide an
> optimal user experience for our use case, a significant amount of custom
> code must be added.
>
> >>> ==Option 5: Re-architect the Repositories== This is the option I
> >>> personally favor for long term. ...
>
> OK, it seems we have agreement, at least at broad scope, that the long-term
> solution will be along the lines of Option 5 (i.e., encourage or enforce
> external hosting; allow for a scenario involving several different repositories).
>
> > here is the route I intend to take:
> > 1. Move the services to a newer more modern host at the ASF
> > (temporary) 2. BandAid the installation to stabilise it for the short
> > term (this is still more work than it sounds)
>
> So it seems we agree on these steps, and it's great (for planning Step
> 1) that you have access to information that is not available to me.
What info would you like access to?
>
> > 3. Stick Apache TrafficServer in front (not varnish) to improve response
> times / caching.
>
> I don't have enough knowledge on Apache TrafficServer to comment on this
> specific step.
Fairly new to me to, but we do have a whole projects worth of committers willing
to help :)
>
> > 4. Go with the choice of Option 5. that is, to allow the hosting and
> downloading of the templates
> > and extensions to be with the 3rd party authors.
>
> It's already allowed; it is just not enforced. I mean, it already happens that
> some extensions form the Extensions site are downloaded from external
> sites like SourceForge.
Ok even better.
>
> > the status quo can not continue, for benefit of all.
> > Help welcomed at any step of the way.
>
> Just make sure to get permission to transfer and modify the site code from
> Oracle, or clarify that such a permission is not needed. I can't have any active
> involvement with this until this issue is solved, even though I completely
> share your desire to bring the sites back to normal operations as soon as
> possible.
Ok sure, not sure we need any permission at this stage but who shall we ask anyway,
Andrew?
(The site will continue to be served from the openoffice.org domain , which is now
ASF owned. I assume you means custom code.)
Gav...
>
> Regards,
> Andrea.
Re: Extensions and templates
Posted by Andrea Pescetti <pe...@openoffice.org>.
On 09/12/2011 Jürgen Schmidt wrote:
> On 12/9/11 1:06 AM, Gavin McDonald wrote:
>> I already have access and have been speaking with Lance over this over
>> the past week. It is in hand, as they say.
This is great.
> One to-do is definitely the upgrade to a newer Drupal version.
I completely agree, but I see this as a medium-term TODO, but not as the
first thing, since upgrading from a heavily customized Drupal 5 to
Drupal 7 is tedious and difficult and the most urgent issues are in
reachability and user handling.
For those who are not following the thread about legacy @openoffice.org
addresses: the Extensions site migration will need to be coordinated
with the strategy for @openoffice.org addresses, see
http://s.apache.org/l7
http://s.apache.org/vMW
and the discussion around them.
Regards,
Andrea.
RE: Extensions and templates
Posted by Gavin McDonald <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Jürgen Schmidt [mailto:jogischmidt@googlemail.com]
> Sent: Friday, 9 December 2011 7:16 PM
> To: ooo-dev@incubator.apache.org
> Subject: Re: Extensions and templates
>
> On 12/9/11 1:06 AM, Gavin McDonald wrote:
> > Dave,
> >
> > I already have access and have been speaking with Lance over this over
> > the past week.
> >
> > It is in hand, as they say.
>
> Gavin, when you have it under control i would like to get access to it to
take a
> closer look on the internals, means the Drupal stuff. I hope i can find
some
> time to dive a little bit deeper into this stuff.
>
> One to-do is definitely the upgrade to a newer Drupal version. I can't
promise
> anything but i will at least try to find the time...
sounds great, will create you an acct over the next few days.
>
> But i have little experience with hosting and deployment of stable, secure
> running web apps. I can only help here and need help of more experienced
> developers in this area.
No problem
Gav...
>
> Juergen
>
>
> >
> > Gav...
> >
> >
> >> -----Original Message-----
> >> From: Dave Fisher [mailto:dave2wave@comcast.net]
> >> Sent: Friday, 9 December 2011 10:03 AM
> >> To: ooo-dev@incubator.apache.org
> >> Subject: Re: Extensions and templates
> >>
> >>
> >> On Dec 8, 2011, at 4:00 AM, Andrea Pescetti wrote:
> >>
> >>> Gavin McDonald wrote:
> >>>>> From: Andrea Pescetti
> >>>>> On 29/11/2011 Rob Weir wrote:
> >>>>>> ==Option 1: Remain at OSUOSL==
> >>>>>> We could remain with OSUOSL hosting. However, the existing site
> >>>>>> is very unstable.
> >>>>> This would be best both for short and long term. ...
> >>>> Sorry, OSUOSL don't want anything to do with these any longer
> >>>
> >>> As I wrote, Options 1 and 3 (i.e., staying with OSUOSL or cloning to
> > another
> >> host) are fundamentally equivalent to me. My point is that the best
> >> first
> > step
> >> is starting from the current websites or a clone of them.
> >>>
> >>>> The hosts themselves cannot cope with all the memory and cpu these
> >>>> are consuming all the time, let alone the bandwidth.
> >>>
> >>> Then someone should explain why they were absolutely stable in 2010,
> >> with a traffic that can be safely assumed to be similar to the one
> >> they
> > are
> >> receiving in 2011. Something broke, and since the Drupal code hasn't
> >> changed I still think that the malfunctioning components are
> >> somewhere
> > else
> >> (or, if in Drupal, not in the site itself but in the interface to
> >> caching
> > engines).
> >>
> >> Send an email to support@osuosl.org and start a conversation with
> >> Lance Albertson. He's willing to tell you all about it. The short
> >> answer is that
> > Oracle
> >> was making changes when the plug was pulled on OOo. They left it
> broken.
> >>
> >> The other part is that the two sites were in such a state that
> >> OSUOSL's
> > Nagios
> >> checks on E&T were like Peter, the boy who cried wolf. They turned
> >> them
> > off
> >> and they don't notice when varnish gets frozen.
> >>
> >> Possibly with a little care you may be able to fix this.
> >>
> >>>> I've had a look around
> >>>> the drupal sites and it is not optimal to say the least.
> >>>
> >>> It would be helpful to know what level of access you obtained to
> >>> make
> > this
> >> assessment. Could you read the site code, or did you receive
> >> administrator credentials for the website, or did you get shell access
to
> the machine?
> >>>
> >>> That the sites are not optimal is fairly obvious, especially
> >>> considering
> > that
> >> Extensions is a Drupal 5 site and thus creates sessions even for
> >> anonymous users; Drupal 7 is much better in this respect and offers
> >> more scalability
> > out
> >> of the box and better integration with caching engines, so it seems a
> > natural
> >> candidate for medium-term and long-term improvements ("Option 5").
> >>
> >> I don't see any reason why you shouldn't ask osuosl.org for access.
> >>
> >> If you and Gavin can both look then we are the correct path to
> >> resolving these troubles. Just agree to work it through!
> >>
> >> Best Regards,
> >> Dave
> >>
> >>
> >>>
> >>>>>> ==Option 2: Move Critical extensions to stable host==
> >>>>> Indeed, as you write, this would be an extreme option.
> >>>> More extreme would be to do nothing, as you'll end up with nothing.
> >>>
> >>> Of course. What I meant to say was: cherry-picking "important"
> > extensions
> >> and creating a repository for them from scratch is more or less the
> >> same work of Option 1 or 3 (i.e., fix the current site or a clone of
> >> it) for a
> > much less
> >> interesting result.
> >>>
> >>>>>> ==Option 3: Clone OSUOSL repositories to another host==
> >>>>> This is not significantly different from Option 1; i.e., if there
> >>>>> are other hosting options available the mere cloning of the site
> >>>>> would not take long, but again the problem is not with the site
> >>>>> but
> > with
> >> caching.
> >>>> Do not blame caches for poor performance. the caches are improving
> >>>> a bad situation
> >>>
> >>> OK, no matter what we think about the root cause for the current bad
> >> performance it seems that we both agree that cloning the site will
> >> give us
> > the
> >> possibility to tweak it (or the environment) and improve performance.
> > Since
> >> I've seen it working perfectly in 2010, I'm confident this is
achievable.
> >>>
> >>>>>> ==Option 4: Host repositories elsewhere, using new UI==
> >>>>> As I used to say, everybody who thinks that the Extensions or
> >>>>> Templates sites can be replaced easily has never tried submitting
> >>>>> a
> >> template!
> >>>>> Thorsten did a lot of customization work on the two sites; any
> >>>>> replacement would provide a largely inferior user experience.
> >>>> I think you don't think very highly of other peoples abilities, a
> >>>> poor
> >> outlook.
> >>>
> >>> That was just a warning: people should know (and they would know, if
> > they
> >> had ever uploaded a template...) that the sites extract and use a lot
> >> of information specific to OpenOffice.org and ODF files. This is
> >> often
> > overlooked
> >> when people see these sites as "some form of file repositories" and
> > propose
> >> to rely on different solutions: they should be aware that, to provide
> >> an optimal user experience for our use case, a significant amount of
> >> custom code must be added.
> >>>
> >>>>>> ==Option 5: Re-architect the Repositories== This is the option I
> >>>>>> personally favor for long term. ...
> >>>
> >>> OK, it seems we have agreement, at least at broad scope, that the
> >>> long-
> >> term solution will be along the lines of Option 5 (i.e., encourage or
> > enforce
> >> external hosting; allow for a scenario involving several different
> > repositories).
> >>>
> >>>> here is the route I intend to take:
> >>>> 1. Move the services to a newer more modern host at the ASF
> >>>> (temporary) 2. BandAid the installation to stabilise it for the
> >>>> short term (this is still more work than it sounds)
> >>>
> >>> So it seems we agree on these steps, and it's great (for planning
> >>> Step
> > 1)
> >> that you have access to information that is not available to me.
> >>>
> >>>> 3. Stick Apache TrafficServer in front (not varnish) to improve
> > response
> >> times / caching.
> >>>
> >>> I don't have enough knowledge on Apache TrafficServer to comment on
> >> this specific step.
> >>>
> >>>> 4. Go with the choice of Option 5. that is, to allow the hosting
> >>>> and
> >> downloading of the templates
> >>>> and extensions to be with the 3rd party authors.
> >>>
> >>> It's already allowed; it is just not enforced. I mean, it already
> > happens that
> >> some extensions form the Extensions site are downloaded from external
> >> sites like SourceForge.
> >>>
> >>>> the status quo can not continue, for benefit of all.
> >>>> Help welcomed at any step of the way.
> >>>
> >>> Just make sure to get permission to transfer and modify the site
> >>> code
> > from
> >> Oracle, or clarify that such a permission is not needed. I can't have
> >> any
> > active
> >> involvement with this until this issue is solved, even though I
> >> completely share your desire to bring the sites back to normal
> >> operations as soon as possible.
> >>>
> >>> Regards,
> >>> Andrea.
> >
> >
Re: Extensions and templates
Posted by Jürgen Schmidt <jo...@googlemail.com>.
On 12/9/11 1:06 AM, Gavin McDonald wrote:
> Dave,
>
> I already have access and have been speaking with Lance over this over the
> past week.
>
> It is in hand, as they say.
Gavin, when you have it under control i would like to get access to it
to take a closer look on the internals, means the Drupal stuff. I hope i
can find some time to dive a little bit deeper into this stuff.
One to-do is definitely the upgrade to a newer Drupal version. I can't
promise anything but i will at least try to find the time...
But i have little experience with hosting and deployment of stable,
secure running web apps. I can only help here and need help of more
experienced developers in this area.
Juergen
>
> Gav...
>
>
>> -----Original Message-----
>> From: Dave Fisher [mailto:dave2wave@comcast.net]
>> Sent: Friday, 9 December 2011 10:03 AM
>> To: ooo-dev@incubator.apache.org
>> Subject: Re: Extensions and templates
>>
>>
>> On Dec 8, 2011, at 4:00 AM, Andrea Pescetti wrote:
>>
>>> Gavin McDonald wrote:
>>>>> From: Andrea Pescetti
>>>>> On 29/11/2011 Rob Weir wrote:
>>>>>> ==Option 1: Remain at OSUOSL==
>>>>>> We could remain with OSUOSL hosting. However, the existing site is
>>>>>> very unstable.
>>>>> This would be best both for short and long term. ...
>>>> Sorry, OSUOSL don't want anything to do with these any longer
>>>
>>> As I wrote, Options 1 and 3 (i.e., staying with OSUOSL or cloning to
> another
>> host) are fundamentally equivalent to me. My point is that the best first
> step
>> is starting from the current websites or a clone of them.
>>>
>>>> The hosts themselves cannot cope with all the memory and cpu these
>>>> are consuming all the time, let alone the bandwidth.
>>>
>>> Then someone should explain why they were absolutely stable in 2010,
>> with a traffic that can be safely assumed to be similar to the one they
> are
>> receiving in 2011. Something broke, and since the Drupal code hasn't
>> changed I still think that the malfunctioning components are somewhere
> else
>> (or, if in Drupal, not in the site itself but in the interface to caching
> engines).
>>
>> Send an email to support@osuosl.org and start a conversation with Lance
>> Albertson. He's willing to tell you all about it. The short answer is that
> Oracle
>> was making changes when the plug was pulled on OOo. They left it broken.
>>
>> The other part is that the two sites were in such a state that OSUOSL's
> Nagios
>> checks on E&T were like Peter, the boy who cried wolf. They turned them
> off
>> and they don't notice when varnish gets frozen.
>>
>> Possibly with a little care you may be able to fix this.
>>
>>>> I've had a look around
>>>> the drupal sites and it is not optimal to say the least.
>>>
>>> It would be helpful to know what level of access you obtained to make
> this
>> assessment. Could you read the site code, or did you receive administrator
>> credentials for the website, or did you get shell access to the machine?
>>>
>>> That the sites are not optimal is fairly obvious, especially considering
> that
>> Extensions is a Drupal 5 site and thus creates sessions even for anonymous
>> users; Drupal 7 is much better in this respect and offers more scalability
> out
>> of the box and better integration with caching engines, so it seems a
> natural
>> candidate for medium-term and long-term improvements ("Option 5").
>>
>> I don't see any reason why you shouldn't ask osuosl.org for access.
>>
>> If you and Gavin can both look then we are the correct path to resolving
>> these troubles. Just agree to work it through!
>>
>> Best Regards,
>> Dave
>>
>>
>>>
>>>>>> ==Option 2: Move Critical extensions to stable host==
>>>>> Indeed, as you write, this would be an extreme option.
>>>> More extreme would be to do nothing, as you'll end up with nothing.
>>>
>>> Of course. What I meant to say was: cherry-picking "important"
> extensions
>> and creating a repository for them from scratch is more or less the same
>> work of Option 1 or 3 (i.e., fix the current site or a clone of it) for a
> much less
>> interesting result.
>>>
>>>>>> ==Option 3: Clone OSUOSL repositories to another host==
>>>>> This is not significantly different from Option 1; i.e., if there
>>>>> are other hosting options available the mere cloning of the site
>>>>> would not take long, but again the problem is not with the site but
> with
>> caching.
>>>> Do not blame caches for poor performance. the caches are improving a
>>>> bad situation
>>>
>>> OK, no matter what we think about the root cause for the current bad
>> performance it seems that we both agree that cloning the site will give us
> the
>> possibility to tweak it (or the environment) and improve performance.
> Since
>> I've seen it working perfectly in 2010, I'm confident this is achievable.
>>>
>>>>>> ==Option 4: Host repositories elsewhere, using new UI==
>>>>> As I used to say, everybody who thinks that the Extensions or
>>>>> Templates sites can be replaced easily has never tried submitting a
>> template!
>>>>> Thorsten did a lot of customization work on the two sites; any
>>>>> replacement would provide a largely inferior user experience.
>>>> I think you don't think very highly of other peoples abilities, a poor
>> outlook.
>>>
>>> That was just a warning: people should know (and they would know, if
> they
>> had ever uploaded a template...) that the sites extract and use a lot of
>> information specific to OpenOffice.org and ODF files. This is often
> overlooked
>> when people see these sites as "some form of file repositories" and
> propose
>> to rely on different solutions: they should be aware that, to provide an
>> optimal user experience for our use case, a significant amount of custom
>> code must be added.
>>>
>>>>>> ==Option 5: Re-architect the Repositories== This is the option I
>>>>>> personally favor for long term. ...
>>>
>>> OK, it seems we have agreement, at least at broad scope, that the long-
>> term solution will be along the lines of Option 5 (i.e., encourage or
> enforce
>> external hosting; allow for a scenario involving several different
> repositories).
>>>
>>>> here is the route I intend to take:
>>>> 1. Move the services to a newer more modern host at the ASF
>>>> (temporary) 2. BandAid the installation to stabilise it for the short
>>>> term (this is still more work than it sounds)
>>>
>>> So it seems we agree on these steps, and it's great (for planning Step
> 1)
>> that you have access to information that is not available to me.
>>>
>>>> 3. Stick Apache TrafficServer in front (not varnish) to improve
> response
>> times / caching.
>>>
>>> I don't have enough knowledge on Apache TrafficServer to comment on
>> this specific step.
>>>
>>>> 4. Go with the choice of Option 5. that is, to allow the hosting and
>> downloading of the templates
>>>> and extensions to be with the 3rd party authors.
>>>
>>> It's already allowed; it is just not enforced. I mean, it already
> happens that
>> some extensions form the Extensions site are downloaded from external
>> sites like SourceForge.
>>>
>>>> the status quo can not continue, for benefit of all.
>>>> Help welcomed at any step of the way.
>>>
>>> Just make sure to get permission to transfer and modify the site code
> from
>> Oracle, or clarify that such a permission is not needed. I can't have any
> active
>> involvement with this until this issue is solved, even though I completely
>> share your desire to bring the sites back to normal operations as soon as
>> possible.
>>>
>>> Regards,
>>> Andrea.
>
>
Re: Extensions and templates
Posted by Dave Fisher <da...@comcast.net>.
On Dec 8, 2011, at 4:35 PM, Dave Fisher wrote:
> Gavin,
>
> On Dec 8, 2011, at 4:06 PM, Gavin McDonald wrote:
>
>> Dave,
>>
>> I already have access and have been speaking with Lance over this over the
>> past week.
>>
>> It is in hand, as they say.
>
> Yes! And we all appreciate this (or should!).
>
> Forums - check, MWiki - check, other wikis - check, Roller - check! You do a lot to support AOO, you have a track record. Thank you, Gavin!
I forgot to mention buildbot support!
Dave
>
> Best Regards,
> Dave
>
>>
>> Gav...
>>
>>
>>> -----Original Message-----
>>> From: Dave Fisher [mailto:dave2wave@comcast.net]
>>> Sent: Friday, 9 December 2011 10:03 AM
>>> To: ooo-dev@incubator.apache.org
>>> Subject: Re: Extensions and templates
>>>
>>>
>>> On Dec 8, 2011, at 4:00 AM, Andrea Pescetti wrote:
>>>
>>>> Gavin McDonald wrote:
>>>>>> From: Andrea Pescetti
>>>>>> On 29/11/2011 Rob Weir wrote:
>>>>>>> ==Option 1: Remain at OSUOSL==
>>>>>>> We could remain with OSUOSL hosting. However, the existing site is
>>>>>>> very unstable.
>>>>>> This would be best both for short and long term. ...
>>>>> Sorry, OSUOSL don't want anything to do with these any longer
>>>>
>>>> As I wrote, Options 1 and 3 (i.e., staying with OSUOSL or cloning to
>> another
>>> host) are fundamentally equivalent to me. My point is that the best first
>> step
>>> is starting from the current websites or a clone of them.
>>>>
>>>>> The hosts themselves cannot cope with all the memory and cpu these
>>>>> are consuming all the time, let alone the bandwidth.
>>>>
>>>> Then someone should explain why they were absolutely stable in 2010,
>>> with a traffic that can be safely assumed to be similar to the one they
>> are
>>> receiving in 2011. Something broke, and since the Drupal code hasn't
>>> changed I still think that the malfunctioning components are somewhere
>> else
>>> (or, if in Drupal, not in the site itself but in the interface to caching
>> engines).
>>>
>>> Send an email to support@osuosl.org and start a conversation with Lance
>>> Albertson. He's willing to tell you all about it. The short answer is that
>> Oracle
>>> was making changes when the plug was pulled on OOo. They left it broken.
>>>
>>> The other part is that the two sites were in such a state that OSUOSL's
>> Nagios
>>> checks on E&T were like Peter, the boy who cried wolf. They turned them
>> off
>>> and they don't notice when varnish gets frozen.
>>>
>>> Possibly with a little care you may be able to fix this.
>>>
>>>>> I've had a look around
>>>>> the drupal sites and it is not optimal to say the least.
>>>>
>>>> It would be helpful to know what level of access you obtained to make
>> this
>>> assessment. Could you read the site code, or did you receive administrator
>>> credentials for the website, or did you get shell access to the machine?
>>>>
>>>> That the sites are not optimal is fairly obvious, especially considering
>> that
>>> Extensions is a Drupal 5 site and thus creates sessions even for anonymous
>>> users; Drupal 7 is much better in this respect and offers more scalability
>> out
>>> of the box and better integration with caching engines, so it seems a
>> natural
>>> candidate for medium-term and long-term improvements ("Option 5").
>>>
>>> I don't see any reason why you shouldn't ask osuosl.org for access.
>>>
>>> If you and Gavin can both look then we are the correct path to resolving
>>> these troubles. Just agree to work it through!
>>>
>>> Best Regards,
>>> Dave
>>>
>>>
>>>>
>>>>>>> ==Option 2: Move Critical extensions to stable host==
>>>>>> Indeed, as you write, this would be an extreme option.
>>>>> More extreme would be to do nothing, as you'll end up with nothing.
>>>>
>>>> Of course. What I meant to say was: cherry-picking "important"
>> extensions
>>> and creating a repository for them from scratch is more or less the same
>>> work of Option 1 or 3 (i.e., fix the current site or a clone of it) for a
>> much less
>>> interesting result.
>>>>
>>>>>>> ==Option 3: Clone OSUOSL repositories to another host==
>>>>>> This is not significantly different from Option 1; i.e., if there
>>>>>> are other hosting options available the mere cloning of the site
>>>>>> would not take long, but again the problem is not with the site but
>> with
>>> caching.
>>>>> Do not blame caches for poor performance. the caches are improving a
>>>>> bad situation
>>>>
>>>> OK, no matter what we think about the root cause for the current bad
>>> performance it seems that we both agree that cloning the site will give us
>> the
>>> possibility to tweak it (or the environment) and improve performance.
>> Since
>>> I've seen it working perfectly in 2010, I'm confident this is achievable.
>>>>
>>>>>>> ==Option 4: Host repositories elsewhere, using new UI==
>>>>>> As I used to say, everybody who thinks that the Extensions or
>>>>>> Templates sites can be replaced easily has never tried submitting a
>>> template!
>>>>>> Thorsten did a lot of customization work on the two sites; any
>>>>>> replacement would provide a largely inferior user experience.
>>>>> I think you don't think very highly of other peoples abilities, a poor
>>> outlook.
>>>>
>>>> That was just a warning: people should know (and they would know, if
>> they
>>> had ever uploaded a template...) that the sites extract and use a lot of
>>> information specific to OpenOffice.org and ODF files. This is often
>> overlooked
>>> when people see these sites as "some form of file repositories" and
>> propose
>>> to rely on different solutions: they should be aware that, to provide an
>>> optimal user experience for our use case, a significant amount of custom
>>> code must be added.
>>>>
>>>>>>> ==Option 5: Re-architect the Repositories== This is the option I
>>>>>>> personally favor for long term. ...
>>>>
>>>> OK, it seems we have agreement, at least at broad scope, that the long-
>>> term solution will be along the lines of Option 5 (i.e., encourage or
>> enforce
>>> external hosting; allow for a scenario involving several different
>> repositories).
>>>>
>>>>> here is the route I intend to take:
>>>>> 1. Move the services to a newer more modern host at the ASF
>>>>> (temporary) 2. BandAid the installation to stabilise it for the short
>>>>> term (this is still more work than it sounds)
>>>>
>>>> So it seems we agree on these steps, and it's great (for planning Step
>> 1)
>>> that you have access to information that is not available to me.
>>>>
>>>>> 3. Stick Apache TrafficServer in front (not varnish) to improve
>> response
>>> times / caching.
>>>>
>>>> I don't have enough knowledge on Apache TrafficServer to comment on
>>> this specific step.
>>>>
>>>>> 4. Go with the choice of Option 5. that is, to allow the hosting and
>>> downloading of the templates
>>>>> and extensions to be with the 3rd party authors.
>>>>
>>>> It's already allowed; it is just not enforced. I mean, it already
>> happens that
>>> some extensions form the Extensions site are downloaded from external
>>> sites like SourceForge.
>>>>
>>>>> the status quo can not continue, for benefit of all.
>>>>> Help welcomed at any step of the way.
>>>>
>>>> Just make sure to get permission to transfer and modify the site code
>> from
>>> Oracle, or clarify that such a permission is not needed. I can't have any
>> active
>>> involvement with this until this issue is solved, even though I completely
>>> share your desire to bring the sites back to normal operations as soon as
>>> possible.
>>>>
>>>> Regards,
>>>> Andrea.
>>
>>
>
Re: Extensions and templates
Posted by Dave Fisher <da...@comcast.net>.
Gavin,
On Dec 8, 2011, at 4:06 PM, Gavin McDonald wrote:
> Dave,
>
> I already have access and have been speaking with Lance over this over the
> past week.
>
> It is in hand, as they say.
Yes! And we all appreciate this (or should!).
Forums - check, MWiki - check, other wikis - check, Roller - check! You do a lot to support AOO, you have a track record. Thank you, Gavin!
Best Regards,
Dave
>
> Gav...
>
>
>> -----Original Message-----
>> From: Dave Fisher [mailto:dave2wave@comcast.net]
>> Sent: Friday, 9 December 2011 10:03 AM
>> To: ooo-dev@incubator.apache.org
>> Subject: Re: Extensions and templates
>>
>>
>> On Dec 8, 2011, at 4:00 AM, Andrea Pescetti wrote:
>>
>>> Gavin McDonald wrote:
>>>>> From: Andrea Pescetti
>>>>> On 29/11/2011 Rob Weir wrote:
>>>>>> ==Option 1: Remain at OSUOSL==
>>>>>> We could remain with OSUOSL hosting. However, the existing site is
>>>>>> very unstable.
>>>>> This would be best both for short and long term. ...
>>>> Sorry, OSUOSL don't want anything to do with these any longer
>>>
>>> As I wrote, Options 1 and 3 (i.e., staying with OSUOSL or cloning to
> another
>> host) are fundamentally equivalent to me. My point is that the best first
> step
>> is starting from the current websites or a clone of them.
>>>
>>>> The hosts themselves cannot cope with all the memory and cpu these
>>>> are consuming all the time, let alone the bandwidth.
>>>
>>> Then someone should explain why they were absolutely stable in 2010,
>> with a traffic that can be safely assumed to be similar to the one they
> are
>> receiving in 2011. Something broke, and since the Drupal code hasn't
>> changed I still think that the malfunctioning components are somewhere
> else
>> (or, if in Drupal, not in the site itself but in the interface to caching
> engines).
>>
>> Send an email to support@osuosl.org and start a conversation with Lance
>> Albertson. He's willing to tell you all about it. The short answer is that
> Oracle
>> was making changes when the plug was pulled on OOo. They left it broken.
>>
>> The other part is that the two sites were in such a state that OSUOSL's
> Nagios
>> checks on E&T were like Peter, the boy who cried wolf. They turned them
> off
>> and they don't notice when varnish gets frozen.
>>
>> Possibly with a little care you may be able to fix this.
>>
>>>> I've had a look around
>>>> the drupal sites and it is not optimal to say the least.
>>>
>>> It would be helpful to know what level of access you obtained to make
> this
>> assessment. Could you read the site code, or did you receive administrator
>> credentials for the website, or did you get shell access to the machine?
>>>
>>> That the sites are not optimal is fairly obvious, especially considering
> that
>> Extensions is a Drupal 5 site and thus creates sessions even for anonymous
>> users; Drupal 7 is much better in this respect and offers more scalability
> out
>> of the box and better integration with caching engines, so it seems a
> natural
>> candidate for medium-term and long-term improvements ("Option 5").
>>
>> I don't see any reason why you shouldn't ask osuosl.org for access.
>>
>> If you and Gavin can both look then we are the correct path to resolving
>> these troubles. Just agree to work it through!
>>
>> Best Regards,
>> Dave
>>
>>
>>>
>>>>>> ==Option 2: Move Critical extensions to stable host==
>>>>> Indeed, as you write, this would be an extreme option.
>>>> More extreme would be to do nothing, as you'll end up with nothing.
>>>
>>> Of course. What I meant to say was: cherry-picking "important"
> extensions
>> and creating a repository for them from scratch is more or less the same
>> work of Option 1 or 3 (i.e., fix the current site or a clone of it) for a
> much less
>> interesting result.
>>>
>>>>>> ==Option 3: Clone OSUOSL repositories to another host==
>>>>> This is not significantly different from Option 1; i.e., if there
>>>>> are other hosting options available the mere cloning of the site
>>>>> would not take long, but again the problem is not with the site but
> with
>> caching.
>>>> Do not blame caches for poor performance. the caches are improving a
>>>> bad situation
>>>
>>> OK, no matter what we think about the root cause for the current bad
>> performance it seems that we both agree that cloning the site will give us
> the
>> possibility to tweak it (or the environment) and improve performance.
> Since
>> I've seen it working perfectly in 2010, I'm confident this is achievable.
>>>
>>>>>> ==Option 4: Host repositories elsewhere, using new UI==
>>>>> As I used to say, everybody who thinks that the Extensions or
>>>>> Templates sites can be replaced easily has never tried submitting a
>> template!
>>>>> Thorsten did a lot of customization work on the two sites; any
>>>>> replacement would provide a largely inferior user experience.
>>>> I think you don't think very highly of other peoples abilities, a poor
>> outlook.
>>>
>>> That was just a warning: people should know (and they would know, if
> they
>> had ever uploaded a template...) that the sites extract and use a lot of
>> information specific to OpenOffice.org and ODF files. This is often
> overlooked
>> when people see these sites as "some form of file repositories" and
> propose
>> to rely on different solutions: they should be aware that, to provide an
>> optimal user experience for our use case, a significant amount of custom
>> code must be added.
>>>
>>>>>> ==Option 5: Re-architect the Repositories== This is the option I
>>>>>> personally favor for long term. ...
>>>
>>> OK, it seems we have agreement, at least at broad scope, that the long-
>> term solution will be along the lines of Option 5 (i.e., encourage or
> enforce
>> external hosting; allow for a scenario involving several different
> repositories).
>>>
>>>> here is the route I intend to take:
>>>> 1. Move the services to a newer more modern host at the ASF
>>>> (temporary) 2. BandAid the installation to stabilise it for the short
>>>> term (this is still more work than it sounds)
>>>
>>> So it seems we agree on these steps, and it's great (for planning Step
> 1)
>> that you have access to information that is not available to me.
>>>
>>>> 3. Stick Apache TrafficServer in front (not varnish) to improve
> response
>> times / caching.
>>>
>>> I don't have enough knowledge on Apache TrafficServer to comment on
>> this specific step.
>>>
>>>> 4. Go with the choice of Option 5. that is, to allow the hosting and
>> downloading of the templates
>>>> and extensions to be with the 3rd party authors.
>>>
>>> It's already allowed; it is just not enforced. I mean, it already
> happens that
>> some extensions form the Extensions site are downloaded from external
>> sites like SourceForge.
>>>
>>>> the status quo can not continue, for benefit of all.
>>>> Help welcomed at any step of the way.
>>>
>>> Just make sure to get permission to transfer and modify the site code
> from
>> Oracle, or clarify that such a permission is not needed. I can't have any
> active
>> involvement with this until this issue is solved, even though I completely
>> share your desire to bring the sites back to normal operations as soon as
>> possible.
>>>
>>> Regards,
>>> Andrea.
>
>
RE: Extensions and templates
Posted by Gavin McDonald <ga...@16degrees.com.au>.
Dave,
I already have access and have been speaking with Lance over this over the
past week.
It is in hand, as they say.
Gav...
> -----Original Message-----
> From: Dave Fisher [mailto:dave2wave@comcast.net]
> Sent: Friday, 9 December 2011 10:03 AM
> To: ooo-dev@incubator.apache.org
> Subject: Re: Extensions and templates
>
>
> On Dec 8, 2011, at 4:00 AM, Andrea Pescetti wrote:
>
> > Gavin McDonald wrote:
> >>> From: Andrea Pescetti
> >>> On 29/11/2011 Rob Weir wrote:
> >>>> ==Option 1: Remain at OSUOSL==
> >>>> We could remain with OSUOSL hosting. However, the existing site is
> >>>> very unstable.
> >>> This would be best both for short and long term. ...
> >> Sorry, OSUOSL don't want anything to do with these any longer
> >
> > As I wrote, Options 1 and 3 (i.e., staying with OSUOSL or cloning to
another
> host) are fundamentally equivalent to me. My point is that the best first
step
> is starting from the current websites or a clone of them.
> >
> >> The hosts themselves cannot cope with all the memory and cpu these
> >> are consuming all the time, let alone the bandwidth.
> >
> > Then someone should explain why they were absolutely stable in 2010,
> with a traffic that can be safely assumed to be similar to the one they
are
> receiving in 2011. Something broke, and since the Drupal code hasn't
> changed I still think that the malfunctioning components are somewhere
else
> (or, if in Drupal, not in the site itself but in the interface to caching
engines).
>
> Send an email to support@osuosl.org and start a conversation with Lance
> Albertson. He's willing to tell you all about it. The short answer is that
Oracle
> was making changes when the plug was pulled on OOo. They left it broken.
>
> The other part is that the two sites were in such a state that OSUOSL's
Nagios
> checks on E&T were like Peter, the boy who cried wolf. They turned them
off
> and they don't notice when varnish gets frozen.
>
> Possibly with a little care you may be able to fix this.
>
> >> I've had a look around
> >> the drupal sites and it is not optimal to say the least.
> >
> > It would be helpful to know what level of access you obtained to make
this
> assessment. Could you read the site code, or did you receive administrator
> credentials for the website, or did you get shell access to the machine?
> >
> > That the sites are not optimal is fairly obvious, especially considering
that
> Extensions is a Drupal 5 site and thus creates sessions even for anonymous
> users; Drupal 7 is much better in this respect and offers more scalability
out
> of the box and better integration with caching engines, so it seems a
natural
> candidate for medium-term and long-term improvements ("Option 5").
>
> I don't see any reason why you shouldn't ask osuosl.org for access.
>
> If you and Gavin can both look then we are the correct path to resolving
> these troubles. Just agree to work it through!
>
> Best Regards,
> Dave
>
>
> >
> >>>> ==Option 2: Move Critical extensions to stable host==
> >>> Indeed, as you write, this would be an extreme option.
> >> More extreme would be to do nothing, as you'll end up with nothing.
> >
> > Of course. What I meant to say was: cherry-picking "important"
extensions
> and creating a repository for them from scratch is more or less the same
> work of Option 1 or 3 (i.e., fix the current site or a clone of it) for a
much less
> interesting result.
> >
> >>>> ==Option 3: Clone OSUOSL repositories to another host==
> >>> This is not significantly different from Option 1; i.e., if there
> >>> are other hosting options available the mere cloning of the site
> >>> would not take long, but again the problem is not with the site but
with
> caching.
> >> Do not blame caches for poor performance. the caches are improving a
> >> bad situation
> >
> > OK, no matter what we think about the root cause for the current bad
> performance it seems that we both agree that cloning the site will give us
the
> possibility to tweak it (or the environment) and improve performance.
Since
> I've seen it working perfectly in 2010, I'm confident this is achievable.
> >
> >>>> ==Option 4: Host repositories elsewhere, using new UI==
> >>> As I used to say, everybody who thinks that the Extensions or
> >>> Templates sites can be replaced easily has never tried submitting a
> template!
> >>> Thorsten did a lot of customization work on the two sites; any
> >>> replacement would provide a largely inferior user experience.
> >> I think you don't think very highly of other peoples abilities, a poor
> outlook.
> >
> > That was just a warning: people should know (and they would know, if
they
> had ever uploaded a template...) that the sites extract and use a lot of
> information specific to OpenOffice.org and ODF files. This is often
overlooked
> when people see these sites as "some form of file repositories" and
propose
> to rely on different solutions: they should be aware that, to provide an
> optimal user experience for our use case, a significant amount of custom
> code must be added.
> >
> >>>> ==Option 5: Re-architect the Repositories== This is the option I
> >>>> personally favor for long term. ...
> >
> > OK, it seems we have agreement, at least at broad scope, that the long-
> term solution will be along the lines of Option 5 (i.e., encourage or
enforce
> external hosting; allow for a scenario involving several different
repositories).
> >
> >> here is the route I intend to take:
> >> 1. Move the services to a newer more modern host at the ASF
> >> (temporary) 2. BandAid the installation to stabilise it for the short
> >> term (this is still more work than it sounds)
> >
> > So it seems we agree on these steps, and it's great (for planning Step
1)
> that you have access to information that is not available to me.
> >
> >> 3. Stick Apache TrafficServer in front (not varnish) to improve
response
> times / caching.
> >
> > I don't have enough knowledge on Apache TrafficServer to comment on
> this specific step.
> >
> >> 4. Go with the choice of Option 5. that is, to allow the hosting and
> downloading of the templates
> >> and extensions to be with the 3rd party authors.
> >
> > It's already allowed; it is just not enforced. I mean, it already
happens that
> some extensions form the Extensions site are downloaded from external
> sites like SourceForge.
> >
> >> the status quo can not continue, for benefit of all.
> >> Help welcomed at any step of the way.
> >
> > Just make sure to get permission to transfer and modify the site code
from
> Oracle, or clarify that such a permission is not needed. I can't have any
active
> involvement with this until this issue is solved, even though I completely
> share your desire to bring the sites back to normal operations as soon as
> possible.
> >
> > Regards,
> > Andrea.
Re: Extensions and templates
Posted by Andrea Pescetti <pe...@openoffice.org>.
Gavin McDonald wrote:
> You should have read my reply to the above email first.
> Please do not bother Lance or OSUOSL with this again, I have access
> and I have it in hand.
I had read your reply of course, but having it "in hand" does not imply
you know all history, and I was interested in knowing what happened.
Anyway, I won't expect to receive answers from OSUOSL then, that's fine.
Regards,
Andrea.
RE: Extensions and templates
Posted by Gavin McDonald <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Andrea Pescetti [mailto:pescetti@openoffice.org]
> Sent: Monday, 12 December 2011 7:51 AM
> To: ooo-dev@incubator.apache.org
> Subject: Re: Extensions and templates
>
> On 09/12/2011 Dave Fisher wrote:
> > Send an email to support@osuosl.org and start a conversation with
> > Lance Albertson. He's willing to tell you all about it. The short
> > answer is that Oracle was making changes when the plug was pulled on
> > OOo. They left it broken.
>
> Done. I hope that I will be able to share some details publicly, otherwise
> everybody will have to e-mail Lance!
>
You should have read my reply to the above email first.
Please do not bother Lance or OSUOSL with this again, I have access
and I have it in hand.
Will update on progress as it happens.
Gav...
> Regards,
> Andrea.
Re: Extensions and templates
Posted by Andrea Pescetti <pe...@openoffice.org>.
On 09/12/2011 Dave Fisher wrote:
> Send an email to support@osuosl.org and start a conversation with
> Lance Albertson. He's willing to tell you all about it. The short
> answer is that Oracle was making changes when the plug was pulled on
> OOo. They left it broken.
Done. I hope that I will be able to share some details publicly,
otherwise everybody will have to e-mail Lance!
Regards,
Andrea.
Re: Extensions and templates
Posted by Dave Fisher <da...@comcast.net>.
On Dec 8, 2011, at 4:00 AM, Andrea Pescetti wrote:
> Gavin McDonald wrote:
>>> From: Andrea Pescetti
>>> On 29/11/2011 Rob Weir wrote:
>>>> ==Option 1: Remain at OSUOSL==
>>>> We could remain with OSUOSL hosting. However, the existing site is
>>>> very unstable.
>>> This would be best both for short and long term. ...
>> Sorry, OSUOSL don’t want anything to do with these any longer
>
> As I wrote, Options 1 and 3 (i.e., staying with OSUOSL or cloning to another host) are fundamentally equivalent to me. My point is that the best first step is starting from the current websites or a clone of them.
>
>> The hosts themselves cannot cope with all the
>> memory and cpu these are consuming all the time, let alone the bandwidth.
>
> Then someone should explain why they were absolutely stable in 2010, with a traffic that can be safely assumed to be similar to the one they are receiving in 2011. Something broke, and since the Drupal code hasn't changed I still think that the malfunctioning components are somewhere else (or, if in Drupal, not in the site itself but in the interface to caching engines).
Send an email to support@osuosl.org and start a conversation with Lance Albertson. He's willing to tell you all about it. The short answer is that Oracle was making changes when the plug was pulled on OOo. They left it broken.
The other part is that the two sites were in such a state that OSUOSL's Nagios checks on E&T were like Peter, the boy who cried wolf. They turned them off and they don't notice when varnish gets frozen.
Possibly with a little care you may be able to fix this.
>> I've had a look around
>> the drupal sites and it is not optimal to say the least.
>
> It would be helpful to know what level of access you obtained to make this assessment. Could you read the site code, or did you receive administrator credentials for the website, or did you get shell access to the machine?
>
> That the sites are not optimal is fairly obvious, especially considering that Extensions is a Drupal 5 site and thus creates sessions even for anonymous users; Drupal 7 is much better in this respect and offers more scalability out of the box and better integration with caching engines, so it seems a natural candidate for medium-term and long-term improvements ("Option 5").
I don't see any reason why you shouldn't ask osuosl.org for access.
If you and Gavin can both look then we are the correct path to resolving these troubles. Just agree to work it through!
Best Regards,
Dave
>
>>>> ==Option 2: Move Critical extensions to stable host==
>>> Indeed, as you write, this would be an extreme option.
>> More extreme would be to do nothing, as you'll end up with nothing.
>
> Of course. What I meant to say was: cherry-picking "important" extensions and creating a repository for them from scratch is more or less the same work of Option 1 or 3 (i.e., fix the current site or a clone of it) for a much less interesting result.
>
>>>> ==Option 3: Clone OSUOSL repositories to another host==
>>> This is not significantly different from Option 1; i.e., if there are other hosting
>>> options available the mere cloning of the site would not take long, but again
>>> the problem is not with the site but with caching.
>> Do not blame caches for poor performance. the caches are improving a bad situation
>
> OK, no matter what we think about the root cause for the current bad performance it seems that we both agree that cloning the site will give us the possibility to tweak it (or the environment) and improve performance. Since I've seen it working perfectly in 2010, I'm confident this is achievable.
>
>>>> ==Option 4: Host repositories elsewhere, using new UI==
>>> As I used to say, everybody who thinks that the Extensions or Templates
>>> sites can be replaced easily has never tried submitting a template!
>>> Thorsten did a lot of customization work on the two sites; any replacement
>>> would provide a largely inferior user experience.
>> I think you don’t think very highly of other peoples abilities, a poor outlook.
>
> That was just a warning: people should know (and they would know, if they had ever uploaded a template...) that the sites extract and use a lot of information specific to OpenOffice.org and ODF files. This is often overlooked when people see these sites as "some form of file repositories" and propose to rely on different solutions: they should be aware that, to provide an optimal user experience for our use case, a significant amount of custom code must be added.
>
>>>> ==Option 5: Re-architect the Repositories== This is the option I
>>>> personally favor for long term. ...
>
> OK, it seems we have agreement, at least at broad scope, that the long-term solution will be along the lines of Option 5 (i.e., encourage or enforce external hosting; allow for a scenario involving several different repositories).
>
>> here is the route I intend to take:
>> 1. Move the services to a newer more modern host at the ASF (temporary)
>> 2. BandAid the installation to stabilise it for the short term (this is still more work than it sounds)
>
> So it seems we agree on these steps, and it's great (for planning Step 1) that you have access to information that is not available to me.
>
>> 3. Stick Apache TrafficServer in front (not varnish) to improve response times / caching.
>
> I don't have enough knowledge on Apache TrafficServer to comment on this specific step.
>
>> 4. Go with the choice of Option 5. that is, to allow the hosting and downloading of the templates
>> and extensions to be with the 3rd party authors.
>
> It's already allowed; it is just not enforced. I mean, it already happens that some extensions form the Extensions site are downloaded from external sites like SourceForge.
>
>> the status quo can not continue, for benefit of all.
>> Help welcomed at any step of the way.
>
> Just make sure to get permission to transfer and modify the site code from Oracle, or clarify that such a permission is not needed. I can't have any active involvement with this until this issue is solved, even though I completely share your desire to bring the sites back to normal operations as soon as possible.
>
> Regards,
> Andrea.
Re: Extensions and templates
Posted by Dave Fisher <da...@comcast.net>.
Andrea / Gavin,
On Dec 12, 2011, at 8:55 PM, Dave Fisher wrote:
>
> On Dec 11, 2011, at 1:52 PM, Andrea Pescetti wrote:
>
>> On 10/12/2011 Gavin McDonald wrote:
>>>> From: Ross Gardler
>>>> Can anyone state that we are legally able to take the
>>>> Drupal code? ...
>>> I checked, got a very clear 'no problem, no issues, please take it'
>>
>> I assume this is enough for us to start looking at the code, even though I would appreciate a public comment by Oracle about this.
>>
>> So you can count me in, my initial focus being on the following two issues:
>>
>> 1) On the existing site, find out what web services / servers in the openoffice.org infrastructure we depend on for authenticating users, and check that authentication can go on even if the main OOo site is migrated. This must be checked before the main OOo infrastructure is moved to Apache. Shell access (or access to the site code anyway) would be best.
>
> If authentication counts on a URL like openoffice.org then we can leave that IP behind at Oracle. http://openoffice.org already redirects to www.openoffice.org. It would be good to know because it effects DNS.
>
> But we could specify all 137 website subdomains individually, and leave *.oo.o and oo.o pointing as now.
Any thoughts about this idea? We really ought to proceed with the site migration pronto.
Regards,
Dave
>
> Regards,
> Dave
>
>>
>> 2) Once the site clone is ready, disconnect it from the single-sign-on and migrate all accounts by notifying users properly.
>
>>
>> Regards,
>> Andrea.
>
Re: Extensions and templates
Posted by Dave Fisher <da...@comcast.net>.
On Dec 11, 2011, at 1:52 PM, Andrea Pescetti wrote:
> On 10/12/2011 Gavin McDonald wrote:
>>> From: Ross Gardler
>>> Can anyone state that we are legally able to take the
>>> Drupal code? ...
>> I checked, got a very clear 'no problem, no issues, please take it'
>
> I assume this is enough for us to start looking at the code, even though I would appreciate a public comment by Oracle about this.
>
> So you can count me in, my initial focus being on the following two issues:
>
> 1) On the existing site, find out what web services / servers in the openoffice.org infrastructure we depend on for authenticating users, and check that authentication can go on even if the main OOo site is migrated. This must be checked before the main OOo infrastructure is moved to Apache. Shell access (or access to the site code anyway) would be best.
If authentication counts on a URL like openoffice.org then we can leave that IP behind at Oracle. http://openoffice.org already redirects to www.openoffice.org. It would be good to know because it effects DNS.
But we could specify all 137 website subdomains individually, and leave *.oo.o and oo.o pointing as now.
Regards,
Dave
>
> 2) Once the site clone is ready, disconnect it from the single-sign-on and migrate all accounts by notifying users properly.
>
> Regards,
> Andrea.
RE: Extensions and templates
Posted by Gavin McDonald <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Andrea Pescetti [mailto:pescetti@openoffice.org]
> Sent: Monday, 12 December 2011 9:20 AM
> To: ooo-dev@incubator.apache.org
> Subject: Re: Extensions and templates
>
> Gavin McDonald wrote:
> > You don't need a public comment from Oracle, the answer I got was from
> > Oracle.
>
> Thanks for sharing this further bit of information.
>
> I confirm I volunteer for the two tasks I leave below.
Thanks Andrea, will give you a shout when ready.
Gav...
>
> Regards,
> Andrea.
>
> >> So you can count me in, my initial focus being on the following two
> > issues:
> >>
> >> 1) On the existing site, find out what web services / servers in the
> >> openoffice.org infrastructure we depend on for authenticating users,
> >> and check that authentication can go on even if the main OOo site is
> migrated.
> >> This must be checked before the main OOo infrastructure is moved to
> >> Apache. Shell access (or access to the site code anyway) would be best.
> >>
> >> 2) Once the site clone is ready, disconnect it from the
> >> single-sign-on and migrate all accounts by notifying users properly.
Re: Extensions and templates
Posted by Andrea Pescetti <pe...@openoffice.org>.
Gavin McDonald wrote:
> You don't need a public comment from Oracle, the answer I got was from
> Oracle.
Thanks for sharing this further bit of information.
I confirm I volunteer for the two tasks I leave below.
Regards,
Andrea.
>> So you can count me in, my initial focus being on the following two
> issues:
>>
>> 1) On the existing site, find out what web services / servers in the
>> openoffice.org infrastructure we depend on for authenticating users, and
>> check that authentication can go on even if the main OOo site is migrated.
>> This must be checked before the main OOo infrastructure is moved to
>> Apache. Shell access (or access to the site code anyway) would be best.
>>
>> 2) Once the site clone is ready, disconnect it from the single-sign-on and
>> migrate all accounts by notifying users properly.
RE: Extensions and templates
Posted by Gavin McDonald <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Andrea Pescetti [mailto:pescetti@openoffice.org]
> Sent: Monday, 12 December 2011 7:52 AM
> To: ooo-dev@incubator.apache.org
> Subject: Re: Extensions and templates
>
> On 10/12/2011 Gavin McDonald wrote:
> >> From: Ross Gardler
> >> Can anyone state that we are legally able to take the Drupal code?
> >> ...
> > I checked, got a very clear 'no problem, no issues, please take it'
>
> I assume this is enough for us to start looking at the code, even though I
> would appreciate a public comment by Oracle about this.
You don't need a public comment from Oracle, the answer I got was from
Oracle.
Gav...
>
> So you can count me in, my initial focus being on the following two
issues:
>
> 1) On the existing site, find out what web services / servers in the
> openoffice.org infrastructure we depend on for authenticating users, and
> check that authentication can go on even if the main OOo site is migrated.
> This must be checked before the main OOo infrastructure is moved to
> Apache. Shell access (or access to the site code anyway) would be best.
>
> 2) Once the site clone is ready, disconnect it from the single-sign-on and
> migrate all accounts by notifying users properly.
>
> Regards,
> Andrea.
Re: Extensions and templates
Posted by Andrea Pescetti <pe...@openoffice.org>.
On 10/12/2011 Gavin McDonald wrote:
>> From: Ross Gardler
>> Can anyone state that we are legally able to take the
>> Drupal code? ...
> I checked, got a very clear 'no problem, no issues, please take it'
I assume this is enough for us to start looking at the code, even though
I would appreciate a public comment by Oracle about this.
So you can count me in, my initial focus being on the following two issues:
1) On the existing site, find out what web services / servers in the
openoffice.org infrastructure we depend on for authenticating users, and
check that authentication can go on even if the main OOo site is
migrated. This must be checked before the main OOo infrastructure is
moved to Apache. Shell access (or access to the site code anyway) would
be best.
2) Once the site clone is ready, disconnect it from the single-sign-on
and migrate all accounts by notifying users properly.
Regards,
Andrea.
Re: Extensions and templates
Posted by Ross Gardler <rg...@opendirective.com>.
On 10 December 2011 00:53, Gavin McDonald <ga...@16degrees.com.au> wrote:
>
>
>> -----Original Message-----
>> From: Ross Gardler [mailto:rgardler@opendirective.com]
>> Sent: Saturday, 10 December 2011 10:13 AM
>> To: ooo-dev@incubator.apache.org
>> Subject: Re: Extensions and templates
>>
>> On 8 December 2011 12:00, Andrea Pescetti <pe...@openoffice.org>
>> wrote:
>>
>> ...
>>
>> > Just make sure to get permission to transfer and modify the site code
>> > from Oracle, or clarify that such a permission is not needed.
>>
>> This came up again in another conversation and I'm not sure if it has been
>> fully cleared up yet. Can anyone state that we are legally able to take
> the
>> Drupal code?
>>
>> If not can someone pick this up while Gav and the growing team of
>> volunteers on this task look at the technical side of things.
>
> I checked, got a very clear 'no problem, no issues, please take it'
Thanks Gav.
Ross
RE: Extensions and templates
Posted by Gavin McDonald <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Ross Gardler [mailto:rgardler@opendirective.com]
> Sent: Saturday, 10 December 2011 10:13 AM
> To: ooo-dev@incubator.apache.org
> Subject: Re: Extensions and templates
>
> On 8 December 2011 12:00, Andrea Pescetti <pe...@openoffice.org>
> wrote:
>
> ...
>
> > Just make sure to get permission to transfer and modify the site code
> > from Oracle, or clarify that such a permission is not needed.
>
> This came up again in another conversation and I'm not sure if it has been
> fully cleared up yet. Can anyone state that we are legally able to take
the
> Drupal code?
>
> If not can someone pick this up while Gav and the growing team of
> volunteers on this task look at the technical side of things.
I checked, got a very clear 'no problem, no issues, please take it'
Gav...
>
> Ross
>
>
> --
> Ross Gardler (@rgardler)
> Programme Leader (Open Development)
> OpenDirective http://opendirective.com
Re: Extensions and templates
Posted by Ross Gardler <rg...@opendirective.com>.
On 8 December 2011 12:00, Andrea Pescetti <pe...@openoffice.org> wrote:
...
> Just make sure to get permission to transfer and modify the site code from
> Oracle, or clarify that such a permission is not needed.
This came up again in another conversation and I'm not sure if it has
been fully cleared up yet. Can anyone state that we are legally able
to take the Drupal code?
If not can someone pick this up while Gav and the growing team of
volunteers on this task look at the technical side of things.
Ross
--
Ross Gardler (@rgardler)
Programme Leader (Open Development)
OpenDirective http://opendirective.com
Re: Extensions and templates
Posted by Andrea Pescetti <pe...@openoffice.org>.
Gavin McDonald wrote:
>> From: Andrea Pescetti
>> On 29/11/2011 Rob Weir wrote:
>>> ==Option 1: Remain at OSUOSL==
>>> We could remain with OSUOSL hosting. However, the existing site is
>>> very unstable.
>> This would be best both for short and long term. ...
> Sorry, OSUOSL don’t want anything to do with these any longer
As I wrote, Options 1 and 3 (i.e., staying with OSUOSL or cloning to
another host) are fundamentally equivalent to me. My point is that the
best first step is starting from the current websites or a clone of them.
> The hosts themselves cannot cope with all the
> memory and cpu these are consuming all the time, let alone the bandwidth.
Then someone should explain why they were absolutely stable in 2010,
with a traffic that can be safely assumed to be similar to the one they
are receiving in 2011. Something broke, and since the Drupal code hasn't
changed I still think that the malfunctioning components are somewhere
else (or, if in Drupal, not in the site itself but in the interface to
caching engines).
> I've had a look around
> the drupal sites and it is not optimal to say the least.
It would be helpful to know what level of access you obtained to make
this assessment. Could you read the site code, or did you receive
administrator credentials for the website, or did you get shell access
to the machine?
That the sites are not optimal is fairly obvious, especially considering
that Extensions is a Drupal 5 site and thus creates sessions even for
anonymous users; Drupal 7 is much better in this respect and offers more
scalability out of the box and better integration with caching engines,
so it seems a natural candidate for medium-term and long-term
improvements ("Option 5").
>>> ==Option 2: Move Critical extensions to stable host==
>> Indeed, as you write, this would be an extreme option.
> More extreme would be to do nothing, as you'll end up with nothing.
Of course. What I meant to say was: cherry-picking "important"
extensions and creating a repository for them from scratch is more or
less the same work of Option 1 or 3 (i.e., fix the current site or a
clone of it) for a much less interesting result.
>>> ==Option 3: Clone OSUOSL repositories to another host==
>> This is not significantly different from Option 1; i.e., if there are other hosting
>> options available the mere cloning of the site would not take long, but again
>> the problem is not with the site but with caching.
> Do not blame caches for poor performance. the caches are improving a bad situation
OK, no matter what we think about the root cause for the current bad
performance it seems that we both agree that cloning the site will give
us the possibility to tweak it (or the environment) and improve
performance. Since I've seen it working perfectly in 2010, I'm confident
this is achievable.
>>> ==Option 4: Host repositories elsewhere, using new UI==
>> As I used to say, everybody who thinks that the Extensions or Templates
>> sites can be replaced easily has never tried submitting a template!
>> Thorsten did a lot of customization work on the two sites; any replacement
>> would provide a largely inferior user experience.
> I think you don’t think very highly of other peoples abilities, a poor outlook.
That was just a warning: people should know (and they would know, if
they had ever uploaded a template...) that the sites extract and use a
lot of information specific to OpenOffice.org and ODF files. This is
often overlooked when people see these sites as "some form of file
repositories" and propose to rely on different solutions: they should be
aware that, to provide an optimal user experience for our use case, a
significant amount of custom code must be added.
>>> ==Option 5: Re-architect the Repositories== This is the option I
>>> personally favor for long term. ...
OK, it seems we have agreement, at least at broad scope, that the
long-term solution will be along the lines of Option 5 (i.e., encourage
or enforce external hosting; allow for a scenario involving several
different repositories).
> here is the route I intend to take:
> 1. Move the services to a newer more modern host at the ASF (temporary)
> 2. BandAid the installation to stabilise it for the short term (this is still more work than it sounds)
So it seems we agree on these steps, and it's great (for planning Step
1) that you have access to information that is not available to me.
> 3. Stick Apache TrafficServer in front (not varnish) to improve response times / caching.
I don't have enough knowledge on Apache TrafficServer to comment on this
specific step.
> 4. Go with the choice of Option 5. that is, to allow the hosting and downloading of the templates
> and extensions to be with the 3rd party authors.
It's already allowed; it is just not enforced. I mean, it already
happens that some extensions form the Extensions site are downloaded
from external sites like SourceForge.
> the status quo can not continue, for benefit of all.
> Help welcomed at any step of the way.
Just make sure to get permission to transfer and modify the site code
from Oracle, or clarify that such a permission is not needed. I can't
have any active involvement with this until this issue is solved, even
though I completely share your desire to bring the sites back to normal
operations as soon as possible.
Regards,
Andrea.
Re: Extensions and templates
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Gavin McDonald wrote on Thu, Dec 08, 2011 at 09:50:33 +1000:
> Help welcomed at any step of the way.
Re: Extensions and templates
Posted by Jürgen Schmidt <jo...@googlemail.com>.
On 12/8/11 10:07 AM, Gavin McDonald wrote:
>
>
>> -----Original Message-----
>> From: Jürgen Schmidt [mailto:jogischmidt@googlemail.com]
>> Sent: Thursday, 8 December 2011 6:40 PM
>> To: ooo-dev@incubator.apache.org
>> Subject: Re: Extensions and templates
>>
>> On 12/8/11 12:50 AM, Gavin McDonald wrote:
>>>
>>>
>>>> -----Original Message-----
>>>> From: Andrea Pescetti [mailto:pescetti@openoffice.org]
>>>> Sent: Thursday, 8 December 2011 8:27 AM
>>>> To: ooo-dev@incubator.apache.org
>>>> Subject: Re: Extensions and templates
>>>>
>>>> On 29/11/2011 Rob Weir wrote:
>>>>> ==Option 1: Remain at OSUOSL==
>>>>> We could remain with OSUOSL hosting. However, the existing site is
>>>>> very unstable.
>>>>
>>>> This would be best both for short and long term. In the short term,
>>>> it provides continuity of service and it doesn't break existing
>>>> links. In the long term, the Drupal instance can be updated and
>>>> extended (it's not easy, but it is something that would have fairly
>>>> high chances of
>>>> success) to get it to be something like the new model (distributed
>>>> repositories) you describe in step 5.
>>>>
>>>
>>> Sorry, OSUOSL don’t want anything to do with these any longer, I
>>> thought folks got the hint when they turned off monitoring and no
>>> longer look at the issues of the host, but rather restart it only when
>>> prompted. The hosts themselves cannot cope with all the memory and cpu
>> these are consuming all the time, let alone the bandwidth.
>>>
>>> This is no longer an option.
>>>
>>>> An important point that everybody seems to miss is that the
>>>> instability of the current Extensions site
>>>> http://extensions.services.openoffice.org/ is, very likely, unrelated
>>>> to Drupal. The underlying Drupal instance is rather sound (and it
>>>> perfectly managed to sustain the traffic in 2010, which should not be
>>>> significantly different from 2011); from the behavior of the site, it
>>>> definitely seems to me that the instability is due to other components,
>> like the caching server (Varnish) in front of it or other caching mechanisms.
>>>
>>> I very much disagree. Caching can be tweaked for sure, but I've had a
>>> look around the drupal sites and it is not optimal to say the least.
>>>
>>>>
>>>> A second very important point is that we need to get the Extensions
>>>> and Templates source code (two different codebases) under the SGA;
>>>> while Drupal itself is GPL, Thorsten Bosbach while working at Oracle
>>>> created a lot of custom PHP code for the two sites. This code, as far
>>>> as I know, has never been released. Access to the source code is a
>>>> prerequisite for any possible analysis/improvement of the website.
>>>>
>>>>> ==Option 2: Move Critical extensions to stable host==
>>>>
>>>> Indeed, as you write, this would be an extreme option.
>>>
>>> More extreme would be to do nothing, as you'll end up with nothing.
>>>
>>>>
>>>>> ==Option 3: Clone OSUOSL repositories to another host==
>>>>
>>>> This is not significantly different from Option 1; i.e., if there are
>>>> other hosting options available the mere cloning of the site would
>>>> not take long, but again the problem is not with the site but with caching.
>>>
>>> Do not blame caches for poor performance. the caches are improving a
>>> bad situation, they can be tweaked to improve further.
>>>
>>>>
>>>> Note that, since the Templates site has already been ported to Drupal
>>>> 6 using the so called "code-driven development" technique, that
>>>> source code would allow to install an empty pre-configured clone of
>>>> the Templates site anywhere. This would be extremely useful for testing.
>>>>
>>>>> ==Option 4: Host repositories elsewhere, using new UI==
>>>>
>>>> As I used to say, everybody who thinks that the Extensions or
>>>> Templates sites can be replaced easily has never tried submitting a
>> template!
>>>> Thorsten did a lot of customization work on the two sites; any
>>>> replacement would provide a largely inferior user experience.
>>>
>>> I think you don’t think very highly of other peoples abilities, a poor outlook.
>>>
>>>>
>>>>> ==Option 5: Re-architect the Repositories== This is the option I
>>>>> personally favor for long term. ...
>>>>> This would allow multiple
>>>>> repositories to look and behave identically from the data perspective.
>>>>
>>>> This is an interesting long term solution indeed, but I see it
>>>> feasible as a
>>>> (complex) version of Option 1-3; i.e., we obtain the current codebase
>>>> with the aim of updating it and extending it in this direction.
>>>>
>>>>> The other thing this approach does is separate the extension
>>>>> metadata from the actual licensed extension. If we wanted to have a
>>>>> canonical repository of registered extensions, but without actually
>>>>> hosting or storing the extensions, then that should be OK. We're
>>>>> hosting URL's to resources. We're not distributing code.
>>>>
>>>> This would offer some advantages, but I see advantages in offering
>>>> hosting for extensions too. The current Extensions site offers both
>>>> options (host there or externally), but if I recall correctly some
>>>> automatic mechanisms, like autogeneration of the update URL, only work
>> if the local hosting is used.
>>>>
>>>
>>> Having spoken to OSUOSL, having looked around the machines and
>>> services in question and having looked at and been told of the
>>> excessive bandwidth (and that is MUST stop), here is the route I intend to
>> take:
>>>
>>> 1. Move the services to a newer more modern host at the ASF
>>> (temporary) 2. BandAid the installation to stabilise it for the short
>>> term (this is still more work than it sounds) 3. Stick Apache TrafficServer in
>> front (not varnish) to improve response times / caching.
>>> 4. Go with the choice of Option 5. that is, to allow the hosting and
>> downloading of the templates
>>> and extensions to be with the 3rd party authors. We will hold master
>> copies, and provide metadata
>>> and links to the download locations / master sites, but we will not allow
>> downloading directly.
>>> This will solve the excessive bandwidth issues longer term. I intend to
>> start the work of this sometime
>>> in January.
>>
>> we should keep in mind that not every extension or template developer has
>> the opportunity to host the extension or template themselves.
>>
>> And then we have potentially the problem that the third party sides are not
>> available when users try to download. Very poor user experience. Ok the
>> moment it is also bad but we are looking for a long term solution.
>>
>> If Apache is not able to host such a repository, we can of course think of
>> multiple repositories in the future.
>>
>> The Apache one would be the default. And here we can host extension that
>> are Apache conform and potentially hosted on our svn or apache-extras.
>>
>>>
>>> If you or anyone else here has any complaints or issues or further
>>> idea, please bring them to the infra team now as I intend to get cracking in
>> this very soon, the status quo can not continue, for benefit of all.
>>>
>>> Help welcomed at any step of the way.
>>>
>>> (Note that moving services from OSUOSL hosts to the ASF hosts does
>>> nothing to solved the bandwidth issues because the ASF servers are
>>> also OSUOSL hosted!)
>>
>> ups, that is not really promising when i think of
>> - a future download of OOo binaries.
>> - the svn performance
>> - the wiki performance (confluence wiki)
>
> That was not the most encouraging comment you could provide this thread
> considering the work I've just volunteered to do to resolve this mess.
i don't know why you feel offended, at least it seems so. It's nothing
personal and I very much appreciate your work. And I hope others
appreciate my work (mainly on the code at the moment) as well.
>
> The overall bandwidth at OSUOSl is not in jeopardy but they are also not
> infinite, one must use them wisely.
>
> Please refrain from being negative.
i don't want to be too negative it's simply my personal impression and
observation (at least the performance).
Juergen
RE: Extensions and templates
Posted by Gavin McDonald <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Jürgen Schmidt [mailto:jogischmidt@googlemail.com]
> Sent: Thursday, 8 December 2011 6:40 PM
> To: ooo-dev@incubator.apache.org
> Subject: Re: Extensions and templates
>
> On 12/8/11 12:50 AM, Gavin McDonald wrote:
> >
> >
> >> -----Original Message-----
> >> From: Andrea Pescetti [mailto:pescetti@openoffice.org]
> >> Sent: Thursday, 8 December 2011 8:27 AM
> >> To: ooo-dev@incubator.apache.org
> >> Subject: Re: Extensions and templates
> >>
> >> On 29/11/2011 Rob Weir wrote:
> >>> ==Option 1: Remain at OSUOSL==
> >>> We could remain with OSUOSL hosting. However, the existing site is
> >>> very unstable.
> >>
> >> This would be best both for short and long term. In the short term,
> >> it provides continuity of service and it doesn't break existing
> >> links. In the long term, the Drupal instance can be updated and
> >> extended (it's not easy, but it is something that would have fairly
> >> high chances of
> >> success) to get it to be something like the new model (distributed
> >> repositories) you describe in step 5.
> >>
> >
> > Sorry, OSUOSL don’t want anything to do with these any longer, I
> > thought folks got the hint when they turned off monitoring and no
> > longer look at the issues of the host, but rather restart it only when
> > prompted. The hosts themselves cannot cope with all the memory and cpu
> these are consuming all the time, let alone the bandwidth.
> >
> > This is no longer an option.
> >
> >> An important point that everybody seems to miss is that the
> >> instability of the current Extensions site
> >> http://extensions.services.openoffice.org/ is, very likely, unrelated
> >> to Drupal. The underlying Drupal instance is rather sound (and it
> >> perfectly managed to sustain the traffic in 2010, which should not be
> >> significantly different from 2011); from the behavior of the site, it
> >> definitely seems to me that the instability is due to other components,
> like the caching server (Varnish) in front of it or other caching mechanisms.
> >
> > I very much disagree. Caching can be tweaked for sure, but I've had a
> > look around the drupal sites and it is not optimal to say the least.
> >
> >>
> >> A second very important point is that we need to get the Extensions
> >> and Templates source code (two different codebases) under the SGA;
> >> while Drupal itself is GPL, Thorsten Bosbach while working at Oracle
> >> created a lot of custom PHP code for the two sites. This code, as far
> >> as I know, has never been released. Access to the source code is a
> >> prerequisite for any possible analysis/improvement of the website.
> >>
> >>> ==Option 2: Move Critical extensions to stable host==
> >>
> >> Indeed, as you write, this would be an extreme option.
> >
> > More extreme would be to do nothing, as you'll end up with nothing.
> >
> >>
> >>> ==Option 3: Clone OSUOSL repositories to another host==
> >>
> >> This is not significantly different from Option 1; i.e., if there are
> >> other hosting options available the mere cloning of the site would
> >> not take long, but again the problem is not with the site but with caching.
> >
> > Do not blame caches for poor performance. the caches are improving a
> > bad situation, they can be tweaked to improve further.
> >
> >>
> >> Note that, since the Templates site has already been ported to Drupal
> >> 6 using the so called "code-driven development" technique, that
> >> source code would allow to install an empty pre-configured clone of
> >> the Templates site anywhere. This would be extremely useful for testing.
> >>
> >>> ==Option 4: Host repositories elsewhere, using new UI==
> >>
> >> As I used to say, everybody who thinks that the Extensions or
> >> Templates sites can be replaced easily has never tried submitting a
> template!
> >> Thorsten did a lot of customization work on the two sites; any
> >> replacement would provide a largely inferior user experience.
> >
> > I think you don’t think very highly of other peoples abilities, a poor outlook.
> >
> >>
> >>> ==Option 5: Re-architect the Repositories== This is the option I
> >>> personally favor for long term. ...
> >>> This would allow multiple
> >>> repositories to look and behave identically from the data perspective.
> >>
> >> This is an interesting long term solution indeed, but I see it
> >> feasible as a
> >> (complex) version of Option 1-3; i.e., we obtain the current codebase
> >> with the aim of updating it and extending it in this direction.
> >>
> >>> The other thing this approach does is separate the extension
> >>> metadata from the actual licensed extension. If we wanted to have a
> >>> canonical repository of registered extensions, but without actually
> >>> hosting or storing the extensions, then that should be OK. We're
> >>> hosting URL's to resources. We're not distributing code.
> >>
> >> This would offer some advantages, but I see advantages in offering
> >> hosting for extensions too. The current Extensions site offers both
> >> options (host there or externally), but if I recall correctly some
> >> automatic mechanisms, like autogeneration of the update URL, only work
> if the local hosting is used.
> >>
> >
> > Having spoken to OSUOSL, having looked around the machines and
> > services in question and having looked at and been told of the
> > excessive bandwidth (and that is MUST stop), here is the route I intend to
> take:
> >
> > 1. Move the services to a newer more modern host at the ASF
> > (temporary) 2. BandAid the installation to stabilise it for the short
> > term (this is still more work than it sounds) 3. Stick Apache TrafficServer in
> front (not varnish) to improve response times / caching.
> > 4. Go with the choice of Option 5. that is, to allow the hosting and
> downloading of the templates
> > and extensions to be with the 3rd party authors. We will hold master
> copies, and provide metadata
> > and links to the download locations / master sites, but we will not allow
> downloading directly.
> > This will solve the excessive bandwidth issues longer term. I intend to
> start the work of this sometime
> > in January.
>
> we should keep in mind that not every extension or template developer has
> the opportunity to host the extension or template themselves.
>
> And then we have potentially the problem that the third party sides are not
> available when users try to download. Very poor user experience. Ok the
> moment it is also bad but we are looking for a long term solution.
>
> If Apache is not able to host such a repository, we can of course think of
> multiple repositories in the future.
>
> The Apache one would be the default. And here we can host extension that
> are Apache conform and potentially hosted on our svn or apache-extras.
>
> >
> > If you or anyone else here has any complaints or issues or further
> > idea, please bring them to the infra team now as I intend to get cracking in
> this very soon, the status quo can not continue, for benefit of all.
> >
> > Help welcomed at any step of the way.
> >
> > (Note that moving services from OSUOSL hosts to the ASF hosts does
> > nothing to solved the bandwidth issues because the ASF servers are
> > also OSUOSL hosted!)
>
> ups, that is not really promising when i think of
> - a future download of OOo binaries.
> - the svn performance
> - the wiki performance (confluence wiki)
That was not the most encouraging comment you could provide this thread
considering the work I've just volunteered to do to resolve this mess.
The overall bandwidth at OSUOSl is not in jeopardy but they are also not
infinite, one must use them wisely.
Please refrain from being negative.
Gav...
>
> Juergen
>
> >
> > Gav...
> >
> >
> >> Regards,
> >> Andrea.
> >
Re: Extensions and templates
Posted by Jürgen Schmidt <jo...@googlemail.com>.
On 12/8/11 12:50 AM, Gavin McDonald wrote:
>
>
>> -----Original Message-----
>> From: Andrea Pescetti [mailto:pescetti@openoffice.org]
>> Sent: Thursday, 8 December 2011 8:27 AM
>> To: ooo-dev@incubator.apache.org
>> Subject: Re: Extensions and templates
>>
>> On 29/11/2011 Rob Weir wrote:
>>> ==Option 1: Remain at OSUOSL==
>>> We could remain with OSUOSL hosting. However, the existing site is
>>> very unstable.
>>
>> This would be best both for short and long term. In the short term, it
>> provides continuity of service and it doesn't break existing links. In the long
>> term, the Drupal instance can be updated and extended (it's not easy, but it
>> is something that would have fairly high chances of
>> success) to get it to be something like the new model (distributed
>> repositories) you describe in step 5.
>>
>
> Sorry, OSUOSL don’t want anything to do with these any longer, I thought folks got the
> hint when they turned off monitoring and no longer look at the issues of the host, but
> rather restart it only when prompted. The hosts themselves cannot cope with all the
> memory and cpu these are consuming all the time, let alone the bandwidth.
>
> This is no longer an option.
>
>> An important point that everybody seems to miss is that the instability of the
>> current Extensions site http://extensions.services.openoffice.org/ is, very
>> likely, unrelated to Drupal. The underlying Drupal instance is rather sound
>> (and it perfectly managed to sustain the traffic in 2010, which should not be
>> significantly different from 2011); from the behavior of the site, it definitely
>> seems to me that the instability is due to other components, like the caching
>> server (Varnish) in front of it or other caching mechanisms.
>
> I very much disagree. Caching can be tweaked for sure, but I've had a look around
> the drupal sites and it is not optimal to say the least.
>
>>
>> A second very important point is that we need to get the Extensions and
>> Templates source code (two different codebases) under the SGA; while
>> Drupal itself is GPL, Thorsten Bosbach while working at Oracle created a lot of
>> custom PHP code for the two sites. This code, as far as I know, has never
>> been released. Access to the source code is a prerequisite for any possible
>> analysis/improvement of the website.
>>
>>> ==Option 2: Move Critical extensions to stable host==
>>
>> Indeed, as you write, this would be an extreme option.
>
> More extreme would be to do nothing, as you'll end up with nothing.
>
>>
>>> ==Option 3: Clone OSUOSL repositories to another host==
>>
>> This is not significantly different from Option 1; i.e., if there are other hosting
>> options available the mere cloning of the site would not take long, but again
>> the problem is not with the site but with caching.
>
> Do not blame caches for poor performance. the caches are improving a bad situation,
> they can be tweaked to improve further.
>
>>
>> Note that, since the Templates site has already been ported to Drupal 6 using
>> the so called "code-driven development" technique, that source code would
>> allow to install an empty pre-configured clone of the Templates site
>> anywhere. This would be extremely useful for testing.
>>
>>> ==Option 4: Host repositories elsewhere, using new UI==
>>
>> As I used to say, everybody who thinks that the Extensions or Templates
>> sites can be replaced easily has never tried submitting a template!
>> Thorsten did a lot of customization work on the two sites; any replacement
>> would provide a largely inferior user experience.
>
> I think you don’t think very highly of other peoples abilities, a poor outlook.
>
>>
>>> ==Option 5: Re-architect the Repositories== This is the option I
>>> personally favor for long term. ...
>>> This would allow multiple
>>> repositories to look and behave identically from the data perspective.
>>
>> This is an interesting long term solution indeed, but I see it feasible as a
>> (complex) version of Option 1-3; i.e., we obtain the current codebase with
>> the aim of updating it and extending it in this direction.
>>
>>> The other thing this approach does is separate the extension metadata
>>> from the actual licensed extension. If we wanted to have a canonical
>>> repository of registered extensions, but without actually hosting or
>>> storing the extensions, then that should be OK. We're hosting URL's
>>> to resources. We're not distributing code.
>>
>> This would offer some advantages, but I see advantages in offering hosting
>> for extensions too. The current Extensions site offers both options (host
>> there or externally), but if I recall correctly some automatic mechanisms, like
>> autogeneration of the update URL, only work if the local hosting is used.
>>
>
> Having spoken to OSUOSL, having looked around the machines and services in question
> and having looked at and been told of the excessive bandwidth (and that is MUST stop),
> here is the route I intend to take:
>
> 1. Move the services to a newer more modern host at the ASF (temporary)
> 2. BandAid the installation to stabilise it for the short term (this is still more work than it sounds)
> 3. Stick Apache TrafficServer in front (not varnish) to improve response times / caching.
> 4. Go with the choice of Option 5. that is, to allow the hosting and downloading of the templates
> and extensions to be with the 3rd party authors. We will hold master copies, and provide metadata
> and links to the download locations / master sites, but we will not allow downloading directly.
> This will solve the excessive bandwidth issues longer term. I intend to start the work of this sometime
> in January.
we should keep in mind that not every extension or template developer
has the opportunity to host the extension or template themselves.
And then we have potentially the problem that the third party sides are
not available when users try to download. Very poor user experience. Ok
the moment it is also bad but we are looking for a long term solution.
If Apache is not able to host such a repository, we can of course think
of multiple repositories in the future.
The Apache one would be the default. And here we can host extension that
are Apache conform and potentially hosted on our svn or apache-extras.
>
> If you or anyone else here has any complaints or issues or further idea, please bring them to the infra team now as I
> intend to get cracking in this very soon, the status quo can not continue, for benefit of all.
>
> Help welcomed at any step of the way.
>
> (Note that moving services from OSUOSL hosts to the ASF hosts does nothing to solved the bandwidth
> issues because the ASF servers are also OSUOSL hosted!)
ups, that is not really promising when i think of
- a future download of OOo binaries.
- the svn performance
- the wiki performance (confluence wiki)
Juergen
>
> Gav...
>
>
>> Regards,
>> Andrea.
>
RE: Extensions and templates
Posted by "Dennis E. Hamilton" <or...@apache.org>.
Gavin,
I have been an ungrateful ass. Please forgive me.
I completely failed to recognize your offer as the contribution that it is.
- It is concrete and specific
- It starts with the least that can possibly work
- It takes us off the hook for having to deal with an emergency loss of the existing operation
Good plan.
Thanks. I see nothing that requires adjustment.
- Dennis
-----Original Message-----
From: Gavin McDonald [mailto:gavin@16degrees.com.au]
Sent: Wednesday, December 07, 2011 15:51
To: ooo-dev@incubator.apache.org
Subject: RE: Extensions and templates
[ ... ]
Having spoken to OSUOSL, having looked around the machines and services in question
and having looked at and been told of the excessive bandwidth (and that is MUST stop),
here is the route I intend to take:
1. Move the services to a newer more modern host at the ASF (temporary)
2. BandAid the installation to stabilise it for the short term (this is still more work than it sounds)
3. Stick Apache TrafficServer in front (not varnish) to improve response times / caching.
4. Go with the choice of Option 5. that is, to allow the hosting and downloading of the templates
and extensions to be with the 3rd party authors. We will hold master copies, and provide metadata
and links to the download locations / master sites, but we will not allow downloading directly.
This will solve the excessive bandwidth issues longer term. I intend to start the work of this sometime
in January.
If you or anyone else here has any complaints or issues or further idea, please bring them to the infra team now as I
intend to get cracking in this very soon, the status quo can not continue, for benefit of all.
Help welcomed at any step of the way.
(Note that moving services from OSUOSL hosts to the ASF hosts does nothing to solved the bandwidth
issues because the ASF servers are also OSUOSL hosted!)
Gav...
[ ... ]
RE: Extensions and templates
Posted by Gavin McDonald <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Andrea Pescetti [mailto:pescetti@openoffice.org]
> Sent: Thursday, 8 December 2011 8:27 AM
> To: ooo-dev@incubator.apache.org
> Subject: Re: Extensions and templates
>
> On 29/11/2011 Rob Weir wrote:
> > ==Option 1: Remain at OSUOSL==
> > We could remain with OSUOSL hosting. However, the existing site is
> > very unstable.
>
> This would be best both for short and long term. In the short term, it
> provides continuity of service and it doesn't break existing links. In the long
> term, the Drupal instance can be updated and extended (it's not easy, but it
> is something that would have fairly high chances of
> success) to get it to be something like the new model (distributed
> repositories) you describe in step 5.
>
Sorry, OSUOSL don’t want anything to do with these any longer, I thought folks got the
hint when they turned off monitoring and no longer look at the issues of the host, but
rather restart it only when prompted. The hosts themselves cannot cope with all the
memory and cpu these are consuming all the time, let alone the bandwidth.
This is no longer an option.
> An important point that everybody seems to miss is that the instability of the
> current Extensions site http://extensions.services.openoffice.org/ is, very
> likely, unrelated to Drupal. The underlying Drupal instance is rather sound
> (and it perfectly managed to sustain the traffic in 2010, which should not be
> significantly different from 2011); from the behavior of the site, it definitely
> seems to me that the instability is due to other components, like the caching
> server (Varnish) in front of it or other caching mechanisms.
I very much disagree. Caching can be tweaked for sure, but I've had a look around
the drupal sites and it is not optimal to say the least.
>
> A second very important point is that we need to get the Extensions and
> Templates source code (two different codebases) under the SGA; while
> Drupal itself is GPL, Thorsten Bosbach while working at Oracle created a lot of
> custom PHP code for the two sites. This code, as far as I know, has never
> been released. Access to the source code is a prerequisite for any possible
> analysis/improvement of the website.
>
> > ==Option 2: Move Critical extensions to stable host==
>
> Indeed, as you write, this would be an extreme option.
More extreme would be to do nothing, as you'll end up with nothing.
>
> > ==Option 3: Clone OSUOSL repositories to another host==
>
> This is not significantly different from Option 1; i.e., if there are other hosting
> options available the mere cloning of the site would not take long, but again
> the problem is not with the site but with caching.
Do not blame caches for poor performance. the caches are improving a bad situation,
they can be tweaked to improve further.
>
> Note that, since the Templates site has already been ported to Drupal 6 using
> the so called "code-driven development" technique, that source code would
> allow to install an empty pre-configured clone of the Templates site
> anywhere. This would be extremely useful for testing.
>
> > ==Option 4: Host repositories elsewhere, using new UI==
>
> As I used to say, everybody who thinks that the Extensions or Templates
> sites can be replaced easily has never tried submitting a template!
> Thorsten did a lot of customization work on the two sites; any replacement
> would provide a largely inferior user experience.
I think you don’t think very highly of other peoples abilities, a poor outlook.
>
> > ==Option 5: Re-architect the Repositories== This is the option I
> > personally favor for long term. ...
> > This would allow multiple
> > repositories to look and behave identically from the data perspective.
>
> This is an interesting long term solution indeed, but I see it feasible as a
> (complex) version of Option 1-3; i.e., we obtain the current codebase with
> the aim of updating it and extending it in this direction.
>
> > The other thing this approach does is separate the extension metadata
> > from the actual licensed extension. If we wanted to have a canonical
> > repository of registered extensions, but without actually hosting or
> > storing the extensions, then that should be OK. We're hosting URL's
> > to resources. We're not distributing code.
>
> This would offer some advantages, but I see advantages in offering hosting
> for extensions too. The current Extensions site offers both options (host
> there or externally), but if I recall correctly some automatic mechanisms, like
> autogeneration of the update URL, only work if the local hosting is used.
>
Having spoken to OSUOSL, having looked around the machines and services in question
and having looked at and been told of the excessive bandwidth (and that is MUST stop),
here is the route I intend to take:
1. Move the services to a newer more modern host at the ASF (temporary)
2. BandAid the installation to stabilise it for the short term (this is still more work than it sounds)
3. Stick Apache TrafficServer in front (not varnish) to improve response times / caching.
4. Go with the choice of Option 5. that is, to allow the hosting and downloading of the templates
and extensions to be with the 3rd party authors. We will hold master copies, and provide metadata
and links to the download locations / master sites, but we will not allow downloading directly.
This will solve the excessive bandwidth issues longer term. I intend to start the work of this sometime
in January.
If you or anyone else here has any complaints or issues or further idea, please bring them to the infra team now as I
intend to get cracking in this very soon, the status quo can not continue, for benefit of all.
Help welcomed at any step of the way.
(Note that moving services from OSUOSL hosts to the ASF hosts does nothing to solved the bandwidth
issues because the ASF servers are also OSUOSL hosted!)
Gav...
> Regards,
> Andrea.
Re: Extensions and templates
Posted by Andrea Pescetti <pe...@openoffice.org>.
On 29/11/2011 Rob Weir wrote:
> ==Option 1: Remain at OSUOSL==
> We could remain with OSUOSL hosting. However, the existing site is
> very unstable.
This would be best both for short and long term. In the short term, it
provides continuity of service and it doesn't break existing links. In
the long term, the Drupal instance can be updated and extended (it's not
easy, but it is something that would have fairly high chances of
success) to get it to be something like the new model (distributed
repositories) you describe in step 5.
An important point that everybody seems to miss is that the instability
of the current Extensions site
http://extensions.services.openoffice.org/ is, very likely, unrelated to
Drupal. The underlying Drupal instance is rather sound (and it perfectly
managed to sustain the traffic in 2010, which should not be
significantly different from 2011); from the behavior of the site, it
definitely seems to me that the instability is due to other components,
like the caching server (Varnish) in front of it or other caching
mechanisms.
A second very important point is that we need to get the Extensions and
Templates source code (two different codebases) under the SGA; while
Drupal itself is GPL, Thorsten Bosbach while working at Oracle created a
lot of custom PHP code for the two sites. This code, as far as I know,
has never been released. Access to the source code is a prerequisite for
any possible analysis/improvement of the website.
> ==Option 2: Move Critical extensions to stable host==
Indeed, as you write, this would be an extreme option.
> ==Option 3: Clone OSUOSL repositories to another host==
This is not significantly different from Option 1; i.e., if there are
other hosting options available the mere cloning of the site would not
take long, but again the problem is not with the site but with caching.
Note that, since the Templates site has already been ported to Drupal 6
using the so called "code-driven development" technique, that source
code would allow to install an empty pre-configured clone of the
Templates site anywhere. This would be extremely useful for testing.
> ==Option 4: Host repositories elsewhere, using new UI==
As I used to say, everybody who thinks that the Extensions or Templates
sites can be replaced easily has never tried submitting a template!
Thorsten did a lot of customization work on the two sites; any
replacement would provide a largely inferior user experience.
> ==Option 5: Re-architect the Repositories==
> This is the option I personally favor for long term. ...
> This would allow multiple
> repositories to look and behave identically from the data perspective.
This is an interesting long term solution indeed, but I see it feasible
as a (complex) version of Option 1-3; i.e., we obtain the current
codebase with the aim of updating it and extending it in this direction.
> The other thing this approach does is separate the extension metadata
> from the actual licensed extension. If we wanted to have a canonical
> repository of registered extensions, but without actually hosting or
> storing the extensions, then that should be OK. We're hosting URL's
> to resources. We're not distributing code.
This would offer some advantages, but I see advantages in offering
hosting for extensions too. The current Extensions site offers both
options (host there or externally), but if I recall correctly some
automatic mechanisms, like autogeneration of the update URL, only work
if the local hosting is used.
Regards,
Andrea.
Re: Extensions and templates
Posted by FR web forum <oo...@free.fr>.
>==Option 1: Remain at OSUOSL==
>
>We could remain with OSUOSL hosting. However, the existing site is
>very unstable. For this approach to be practical we'd need a
>volunteer with Drupal skills to work with OSUOSL to diagnose what is
>wrong and to restore stability to these services. Maybe some sight
>maintenance, upgrades or tuning is sufficient?
+1
Drupal is better to manage multilingual UI