You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Paul Querna <pq...@apache.org> on 2007/02/26 17:59:59 UTC
Use of Proxy on directory.apache.org
Hello dev@directory;
Us over in infrastructure noticed something different about
http://directory.apache.org/ this morning. Well, first thing was, that
it was down. (And people noticed, and told us!)
But when we looked into it, we thought it was pretty odd, since eos (the
T2000 that all of the main websites are currently hosted on) was doing
fine, and all the other TLP sites seemed up.
What we found was a cute series of RewriteRules in the main
directory.apache.org .htaccess file:
$ cat /x1/www/directory.apache.org/.htaccess
....
RewriteRule ^(.*)$ http://cwiki.apache.org/DIRxSITE/$1 [P,L]
So, most of the content on directory.apache.org is being proxied to
cwiki.apache.org aka brutus.apache.org.
>From an infrastructure perspective, this creates an unacceptable
dependency between services. brutus.apache.org (and CWiki) will go
down. They just aren't setup to be redundant right now.
In comparison, the (static) content under /x1/www is mirrored to
multiple machines in multiple data centers, and in the case of a failure
we can quickly (and relatively painlessly) switch it to another machine.
The preferred method is to export the static files from brutus, onto
people.apache.org, into your /x1/www/directory.apache.org/ directory.
Many other TLPs do this via a crontab on people.apache.org. If you need
help setting this up, we would be happy to help get you started.
Thanks,
-Paul
Re: Use of Proxy on directory.apache.org
Posted by Emmanuel Lecharny <el...@gmail.com>.
Thanks a lot Ersin for those explanations. I hope that soon we will find a
better solution. Sorry if I can't help, but I think it's better that the
started effort be continued by people who know what it's all about !
Emmanuel
On 3/15/07, Ersin Er <er...@gmail.com> wrote:
>
> Hi all,
>
> Well, a few infra folks and I tried to set this up as suggested during
> an IRC session but we did not succeed. BTW they helped me to write a
> cron job to sync all of our spaces from cwiki to d.a.o/cwiki (you
> cannot see this via your browser). We currently have all the content
> under Directory site but we need some local rewrite rules to make them
> match our current name space. We have made some progress but as the
> Directory project uses many wiki spaces bound to different name spaces
> (under d.a.o) we had trouble with configuring the ReWrite plugin. A
> suggestion was to us the Alias plugin (for httpd) but infra folks
> realized that it was not allowed to use it in .htaccess files. So we
> suspended the migration. I hope we can do one more attempt soon to fix
> the problem. BTW, I think it's quite easy for MINA to migrate to that
> suggested model.
>
> Cheers,
>
> On 3/15/07, Alex Karasulu <ak...@apache.org> wrote:
> >
> >
> >
> > On 3/15/07, Upayavira <uv...@odoko.co.uk> wrote:
> > > Joshua Slive wrote:
> > > > On 3/15/07, Upayavira <uv...@odoko.co.uk> wrote:
> > > >>
> > > >> Someone suggested a symlink from /www to ~jefft. Would that
> actually
> > > >> work?
> > > >
> > > > No. The content would not get properly rsynced to the live site or
> to
> > > > backup-sites.
> > > >
> > > > I don't understand what is the big deal about just copying it to the
> > > > correct place.
> > >
> > > The big deal is that to set up an rsync you need to do that in a
> user's
> > > crontab. That ties control of the process down to a single user, which
> > > is far from preferable.
> >
> >
> > Yes I would have to agree with Upayavira here. It's not a good idea to
> > depend on a single user account for this. Plus that person must be
> hunted
> > down and contacted any time there needs to be some modification done.
> >
> >
> > > The fact that we now have /x1/www/confluence-exports sorts half of the
> > > picture. If symlinks work, that sorts the other half.
> > >
> > > Reasonable?
> >
> >
> > +1
> >
> > Alex
> >
> >
> >
>
>
> --
> Ersin
>
--
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Use of Proxy on directory.apache.org
Posted by Joshua Slive <jo...@slive.ca>.
On 3/15/07, Ersin Er <er...@gmail.com> wrote:
> Hi all,
>
> Well, a few infra folks and I tried to set this up as suggested during
> an IRC session but we did not succeed. BTW they helped me to write a
> cron job to sync all of our spaces from cwiki to d.a.o/cwiki (you
> cannot see this via your browser). We currently have all the content
> under Directory site but we need some local rewrite rules to make them
> match our current name space. We have made some progress but as the
> Directory project uses many wiki spaces bound to different name spaces
> (under d.a.o) we had trouble with configuring the ReWrite plugin. A
> suggestion was to us the Alias plugin (for httpd) but infra folks
> realized that it was not allowed to use it in .htaccess files. So we
> suspended the migration. I hope we can do one more attempt soon to fix
> the problem. BTW, I think it's quite easy for MINA to migrate to that
> suggested model.
Anything that can be done with an Alias can also be done with a
RewriteRule. From peeking into your .htaccess, you can, for example,
replace
Alias /apacheds/1.0 /x1/www/directory.apache.org/cwiki/DIRxSRVx10
with
RewriteRule ^apacheds/1\.0(.*) /x1/www/directory.apache.org/cwiki/DIRxSRVx10$1
But that looks like an over-complex nightmare to me. Why don't you
just change the directory names when you do the rsync copy. In other
words, rsync DIRxSRVx10 directly into
/www/directory.apache.org/apacheds/1.0/.
Joshua.
Re: Use of Proxy on directory.apache.org
Posted by Ersin Er <er...@gmail.com>.
On 3/15/07, Joshua Slive <jo...@slive.ca> wrote:
> On 3/15/07, Ersin Er <er...@gmail.com> wrote:
> > On 3/15/07, Joshua Slive <jo...@slive.ca> wrote:
>
> > > Anything that can be done with an Alias can also be done with a
> > > RewriteRule. From peeking into your .htaccess, you can, for example,
> > > replace
> > > Alias /apacheds/1.0 /x1/www/directory.apache.org/cwiki/DIRxSRVx10
> > > with
> > > RewriteRule ^apacheds/1\.0(.*) /x1/www/directory.apache.org/cwiki/DIRxSRVx10$1
> > >
> > > But that looks like an over-complex nightmare to me. Why don't you
> > > just change the directory names when you do the rsync copy. In other
> > > words, rsync DIRxSRVx10 directly into
> > > /www/directory.apache.org/apacheds/1.0/.
> >
> > The solution is not that simple. (Or I was not able to solve as a more
> > possible option.) To be able to do that we need to change our name
> > space organization. If we directly rsync to final destinations we'll
> > loose control of manually added content (like d.a.o/staticresources or
> > d.a.o/ldapstudio/update).
>
> This is easy enough to solve by either omitting the --delete option of
> rsync (which will cause it to leave existing content untouched) and
> then manually cleaning old stuff occasionally, or using the --exclude
> option of rsync to exclude those directories.
>
> Joshua.
>
OK, we'll give it a try. Thanks for the suggestion.
--
Ersin
Re: Use of Proxy on directory.apache.org
Posted by Joshua Slive <jo...@slive.ca>.
On 3/15/07, Ersin Er <er...@gmail.com> wrote:
> On 3/15/07, Joshua Slive <jo...@slive.ca> wrote:
> > Anything that can be done with an Alias can also be done with a
> > RewriteRule. From peeking into your .htaccess, you can, for example,
> > replace
> > Alias /apacheds/1.0 /x1/www/directory.apache.org/cwiki/DIRxSRVx10
> > with
> > RewriteRule ^apacheds/1\.0(.*) /x1/www/directory.apache.org/cwiki/DIRxSRVx10$1
> >
> > But that looks like an over-complex nightmare to me. Why don't you
> > just change the directory names when you do the rsync copy. In other
> > words, rsync DIRxSRVx10 directly into
> > /www/directory.apache.org/apacheds/1.0/.
>
> The solution is not that simple. (Or I was not able to solve as a more
> possible option.) To be able to do that we need to change our name
> space organization. If we directly rsync to final destinations we'll
> loose control of manually added content (like d.a.o/staticresources or
> d.a.o/ldapstudio/update).
This is easy enough to solve by either omitting the --delete option of
rsync (which will cause it to leave existing content untouched) and
then manually cleaning old stuff occasionally, or using the --exclude
option of rsync to exclude those directories.
Joshua.
Re: Use of Proxy on directory.apache.org
Posted by Ersin Er <er...@gmail.com>.
Hi all,
Well, a few infra folks and I tried to set this up as suggested during
an IRC session but we did not succeed. BTW they helped me to write a
cron job to sync all of our spaces from cwiki to d.a.o/cwiki (you
cannot see this via your browser). We currently have all the content
under Directory site but we need some local rewrite rules to make them
match our current name space. We have made some progress but as the
Directory project uses many wiki spaces bound to different name spaces
(under d.a.o) we had trouble with configuring the ReWrite plugin. A
suggestion was to us the Alias plugin (for httpd) but infra folks
realized that it was not allowed to use it in .htaccess files. So we
suspended the migration. I hope we can do one more attempt soon to fix
the problem. BTW, I think it's quite easy for MINA to migrate to that
suggested model.
Cheers,
On 3/15/07, Alex Karasulu <ak...@apache.org> wrote:
>
>
>
> On 3/15/07, Upayavira <uv...@odoko.co.uk> wrote:
> > Joshua Slive wrote:
> > > On 3/15/07, Upayavira <uv...@odoko.co.uk> wrote:
> > >>
> > >> Someone suggested a symlink from /www to ~jefft. Would that actually
> > >> work?
> > >
> > > No. The content would not get properly rsynced to the live site or to
> > > backup-sites.
> > >
> > > I don't understand what is the big deal about just copying it to the
> > > correct place.
> >
> > The big deal is that to set up an rsync you need to do that in a user's
> > crontab. That ties control of the process down to a single user, which
> > is far from preferable.
>
>
> Yes I would have to agree with Upayavira here. It's not a good idea to
> depend on a single user account for this. Plus that person must be hunted
> down and contacted any time there needs to be some modification done.
>
>
> > The fact that we now have /x1/www/confluence-exports sorts half of the
> > picture. If symlinks work, that sorts the other half.
> >
> > Reasonable?
>
>
> +1
>
> Alex
>
>
>
--
Ersin
Re: Use of Proxy on directory.apache.org
Posted by Alex Karasulu <ak...@apache.org>.
On 3/15/07, Upayavira <uv...@odoko.co.uk> wrote:
>
> Joshua Slive wrote:
> > On 3/15/07, Upayavira <uv...@odoko.co.uk> wrote:
> >>
> >> Someone suggested a symlink from /www to ~jefft. Would that actually
> >> work?
> >
> > No. The content would not get properly rsynced to the live site or to
> > backup-sites.
> >
> > I don't understand what is the big deal about just copying it to the
> > correct place.
>
> The big deal is that to set up an rsync you need to do that in a user's
> crontab. That ties control of the process down to a single user, which
> is far from preferable.
Yes I would have to agree with Upayavira here. It's not a good idea to
depend on a single user account for this. Plus that person must be hunted
down and contacted any time there needs to be some modification done.
The fact that we now have /x1/www/confluence-exports sorts half of the
> picture. If symlinks work, that sorts the other half.
>
> Reasonable?
+1
Alex
Re: Use of Proxy on directory.apache.org
Posted by Joshua Slive <jo...@slive.ca>.
[Fixing reply-to.]
On 3/15/07, Trustin Lee <tr...@gmail.com> wrote:
> If it's already being rsync'd to people.apache.org, we could modify
> our rewrite rule like the following instead of using a crontab:
>
> RewriteRule ^(.*) http://people.apache.org/~jefft/confluence/MINA/$1 [P]
>
> WDYT?
No.
That doesn't help at all, in fact, since www.apache.org and
people.apache.org are not served from the same machine.
Simply take the content that jefft has rsynced, and use a cron job to
put it in your directory under minotaur:/www/
Re: Use of Proxy on directory.apache.org
Posted by Trustin Lee <tr...@gmail.com>.
Hi infra team,
Sorry for late response. I must confess that this issue also affects
mina.apache.org, and it was me who started to use RewriteRule.
On 2/27/07, Paul Querna <pq...@apache.org> wrote:
> Ersin Er wrote:
> > Hi,
> >
> > Mostly I am responsible of those Rewrite Rules. I would be glad to
> > learn and apply the general way of doing this sync stuff. How can we
> > start it? On IRC, or on the list?
>
> Doesn't matter to me...
>
> The content from brutus is already copied over by jefft to this location
> on people.apache.org:
> /x1/home/jefft/public_html/confluence
>
> All you need to do is on people.apache.org, add a crontab to copy the
> content you want from /x1/home/jefft/public_html/confluence to
> /x1/www/directory.apache.org/
If it's already being rsync'd to people.apache.org, we could modify
our rewrite rule like the following instead of using a crontab:
RewriteRule ^(.*) http://people.apache.org/~jefft/confluence/MINA/$1 [P]
WDYT?
Anyways, how often is Jeff's directory being rsync'd?
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: Use of Proxy on directory.apache.org
Posted by Upayavira <uv...@odoko.co.uk>.
Alex Karasulu wrote:
> Might be best to go directly from brutus to people.a.o so our own sync
> frequency can be managed properly. Also we may need to trigger this
> manually at times.
The rsync happens from brutus crontab.
> However access to brutus might be an issue right?
At some point, when I get enough time (!), I'd like to think through a
proper rsync scheme that doesn't require access to a user's personal
crontab.
Regards, Upayavira
Re: Use of Proxy on directory.apache.org
Posted by Alex Karasulu <ak...@apache.org>.
Might be best to go directly from brutus to people.a.o so our own sync
frequency can be managed properly. Also we may need to trigger this
manually at times.
However access to brutus might be an issue right?
Alex
On 2/26/07, Paul Querna <pq...@apache.org> wrote:
>
> Ersin Er wrote:
> > Hi,
> >
> > Mostly I am responsible of those Rewrite Rules. I would be glad to
> > learn and apply the general way of doing this sync stuff. How can we
> > start it? On IRC, or on the list?
>
> Doesn't matter to me...
>
> The content from brutus is already copied over by jefft to this location
> on people.apache.org:
> /x1/home/jefft/public_html/confluence
>
> All you need to do is on people.apache.org, add a crontab to copy the
> content you want from /x1/home/jefft/public_html/confluence to
> /x1/www/directory.apache.org/
>
>
> > On 2/26/07, Alex Karasulu <ak...@apache.org> wrote:
> >> Hi Paul,
> >>
> >> Thanks for bringing this to our attention. I think setting up a
> >> simple cron
> >> job to sync the content over is better than the rewrite rules. We
> >> certainly
> >> would like to have the redundancy and rapid fail over that other sites
> >> enjoy.
> >>
> >> Hopefully we can get a response from some of the folks that were
> >> involved in
> >> setting up the site so we can quickly rectify this situation.
> >>
> >> Thanks,
> >> Alex
> >>
> >>
> >> On 2/26/07, Paul Querna <pq...@apache.org> wrote:
> >> > Hello dev@directory;
> >> >
> >> > Us over in infrastructure noticed something different about
> >> > http://directory.apache.org/ this morning. Well, first thing was,
> that
> >> > it was down. (And people noticed, and told us!)
> >> >
> >> > But when we looked into it, we thought it was pretty odd, since eos
> >> (the
> >> > T2000 that all of the main websites are currently hosted on) was
> doing
> >> > fine, and all the other TLP sites seemed up.
> >> >
> >> > What we found was a cute series of RewriteRules in the main
> >> > directory.apache.org .htaccess file:
> >> > $ cat /x1/www/directory.apache.org/.htaccess
> >> > ....
> >> > RewriteRule ^(.*)$ http://cwiki.apache.org/DIRxSITE/$1
> >> [P,L]
> >> >
> >> >
> >> > So, most of the content on directory.apache.org is being proxied to
> >> > cwiki.apache.org aka brutus.apache.org.
> >> >
> >> > From an infrastructure perspective, this creates an unacceptable
> >> > dependency between services. brutus.apache.org (and CWiki) will go
> >> > down. They just aren't setup to be redundant right now.
> >> >
> >> > In comparison, the (static) content under /x1/www is mirrored to
> >> > multiple machines in multiple data centers, and in the case of a
> >> failure
> >> > we can quickly (and relatively painlessly) switch it to another
> >> machine.
> >> >
> >> > The preferred method is to export the static files from brutus, onto
> >> > people.apache.org, into your /x1/www/directory.apache.org/ directory.
> >> > Many other TLPs do this via a crontab on people.apache.org. If you
> >> need
> >> > help setting this up, we would be happy to help get you started.
> >> >
> >> > Thanks,
> >> >
> >> > -Paul
> >> >
> >>
> >>
> >
> >
>
>
Re: Use of Proxy on directory.apache.org
Posted by Paul Querna <pq...@apache.org>.
Ersin Er wrote:
> Hi,
>
> Mostly I am responsible of those Rewrite Rules. I would be glad to
> learn and apply the general way of doing this sync stuff. How can we
> start it? On IRC, or on the list?
Doesn't matter to me...
The content from brutus is already copied over by jefft to this location
on people.apache.org:
/x1/home/jefft/public_html/confluence
All you need to do is on people.apache.org, add a crontab to copy the
content you want from /x1/home/jefft/public_html/confluence to
/x1/www/directory.apache.org/
> On 2/26/07, Alex Karasulu <ak...@apache.org> wrote:
>> Hi Paul,
>>
>> Thanks for bringing this to our attention. I think setting up a
>> simple cron
>> job to sync the content over is better than the rewrite rules. We
>> certainly
>> would like to have the redundancy and rapid fail over that other sites
>> enjoy.
>>
>> Hopefully we can get a response from some of the folks that were
>> involved in
>> setting up the site so we can quickly rectify this situation.
>>
>> Thanks,
>> Alex
>>
>>
>> On 2/26/07, Paul Querna <pq...@apache.org> wrote:
>> > Hello dev@directory;
>> >
>> > Us over in infrastructure noticed something different about
>> > http://directory.apache.org/ this morning. Well, first thing was, that
>> > it was down. (And people noticed, and told us!)
>> >
>> > But when we looked into it, we thought it was pretty odd, since eos
>> (the
>> > T2000 that all of the main websites are currently hosted on) was doing
>> > fine, and all the other TLP sites seemed up.
>> >
>> > What we found was a cute series of RewriteRules in the main
>> > directory.apache.org .htaccess file:
>> > $ cat /x1/www/directory.apache.org/.htaccess
>> > ....
>> > RewriteRule ^(.*)$ http://cwiki.apache.org/DIRxSITE/$1
>> [P,L]
>> >
>> >
>> > So, most of the content on directory.apache.org is being proxied to
>> > cwiki.apache.org aka brutus.apache.org.
>> >
>> > From an infrastructure perspective, this creates an unacceptable
>> > dependency between services. brutus.apache.org (and CWiki) will go
>> > down. They just aren't setup to be redundant right now.
>> >
>> > In comparison, the (static) content under /x1/www is mirrored to
>> > multiple machines in multiple data centers, and in the case of a
>> failure
>> > we can quickly (and relatively painlessly) switch it to another
>> machine.
>> >
>> > The preferred method is to export the static files from brutus, onto
>> > people.apache.org, into your /x1/www/directory.apache.org/ directory.
>> > Many other TLPs do this via a crontab on people.apache.org. If you
>> need
>> > help setting this up, we would be happy to help get you started.
>> >
>> > Thanks,
>> >
>> > -Paul
>> >
>>
>>
>
>
Re: Use of Proxy on directory.apache.org
Posted by Joe Schaefer <jo...@sunstarsys.com>.
"Ersin Er" <er...@gmail.com> writes:
> Hi,
>
> Mostly I am responsible of those Rewrite Rules. I would be glad to
> learn and apply the general way of doing this sync stuff. How can we
> start it? On IRC, or on the list?
Jeff Turner maintains a confluence sync from cwiki to people here:
http://people.apache.org/~jefft/confluence/
So if you set up a cron that copied
/home/jefft/public_html/confluence/DIRxSITE
to /x1/www/directory.apache.org, that should do the trick.
--
Joe Schaefer
Re: Use of Proxy on directory.apache.org
Posted by Ersin Er <er...@gmail.com>.
Hi,
Mostly I am responsible of those Rewrite Rules. I would be glad to
learn and apply the general way of doing this sync stuff. How can we
start it? On IRC, or on the list?
Thanks.
On 2/26/07, Alex Karasulu <ak...@apache.org> wrote:
> Hi Paul,
>
> Thanks for bringing this to our attention. I think setting up a simple cron
> job to sync the content over is better than the rewrite rules. We certainly
> would like to have the redundancy and rapid fail over that other sites
> enjoy.
>
> Hopefully we can get a response from some of the folks that were involved in
> setting up the site so we can quickly rectify this situation.
>
> Thanks,
> Alex
>
>
> On 2/26/07, Paul Querna <pq...@apache.org> wrote:
> > Hello dev@directory;
> >
> > Us over in infrastructure noticed something different about
> > http://directory.apache.org/ this morning. Well, first thing was, that
> > it was down. (And people noticed, and told us!)
> >
> > But when we looked into it, we thought it was pretty odd, since eos (the
> > T2000 that all of the main websites are currently hosted on) was doing
> > fine, and all the other TLP sites seemed up.
> >
> > What we found was a cute series of RewriteRules in the main
> > directory.apache.org .htaccess file:
> > $ cat /x1/www/directory.apache.org/.htaccess
> > ....
> > RewriteRule ^(.*)$ http://cwiki.apache.org/DIRxSITE/$1
> [P,L]
> >
> >
> > So, most of the content on directory.apache.org is being proxied to
> > cwiki.apache.org aka brutus.apache.org.
> >
> > From an infrastructure perspective, this creates an unacceptable
> > dependency between services. brutus.apache.org (and CWiki) will go
> > down. They just aren't setup to be redundant right now.
> >
> > In comparison, the (static) content under /x1/www is mirrored to
> > multiple machines in multiple data centers, and in the case of a failure
> > we can quickly (and relatively painlessly) switch it to another machine.
> >
> > The preferred method is to export the static files from brutus, onto
> > people.apache.org, into your /x1/www/directory.apache.org/ directory.
> > Many other TLPs do this via a crontab on people.apache.org. If you need
> > help setting this up, we would be happy to help get you started.
> >
> > Thanks,
> >
> > -Paul
> >
>
>
--
Ersin
Re: Use of Proxy on directory.apache.org
Posted by Alex Karasulu <ak...@apache.org>.
Hi Paul,
Thanks for bringing this to our attention. I think setting up a simple cron
job to sync the content over is better than the rewrite rules. We certainly
would like to have the redundancy and rapid fail over that other sites
enjoy.
Hopefully we can get a response from some of the folks that were involved in
setting up the site so we can quickly rectify this situation.
Thanks,
Alex
On 2/26/07, Paul Querna <pq...@apache.org> wrote:
>
> Hello dev@directory;
>
> Us over in infrastructure noticed something different about
> http://directory.apache.org/ this morning. Well, first thing was, that
> it was down. (And people noticed, and told us!)
>
> But when we looked into it, we thought it was pretty odd, since eos (the
> T2000 that all of the main websites are currently hosted on) was doing
> fine, and all the other TLP sites seemed up.
>
> What we found was a cute series of RewriteRules in the main
> directory.apache.org .htaccess file:
> $ cat /x1/www/directory.apache.org/.htaccess
> ....
> RewriteRule ^(.*)$ http://cwiki.apache.org/DIRxSITE/$1 [P,L]
>
>
> So, most of the content on directory.apache.org is being proxied to
> cwiki.apache.org aka brutus.apache.org.
>
> From an infrastructure perspective, this creates an unacceptable
> dependency between services. brutus.apache.org (and CWiki) will go
> down. They just aren't setup to be redundant right now.
>
> In comparison, the (static) content under /x1/www is mirrored to
> multiple machines in multiple data centers, and in the case of a failure
> we can quickly (and relatively painlessly) switch it to another machine.
>
> The preferred method is to export the static files from brutus, onto
> people.apache.org, into your /x1/www/directory.apache.org/ directory.
> Many other TLPs do this via a crontab on people.apache.org. If you need
> help setting this up, we would be happy to help get you started.
>
> Thanks,
>
> -Paul
>