You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Daniel Sahlberg <da...@gmail.com> on 2020/12/21 07:55:11 UTC
Re: svn.haxx.se is going away
Den fre 27 nov. 2020 kl 19:26 skrev Daniel Shahaf <d....@daniel.shahaf.name>:
> Sounds good. Nathan, Daniel Sahlberg — could you work with Infra on
> getting the data over to ASF hardware?
>
I have been given access to svn-qavm and uploaded a tarball of the website
(including mboxes). I'm a bit reluctant to unpack it since it takes almost
7GB, and there is only 14 GB disk space remaining. Is it ok to unpack or
should we ask Infra for more disk space?
Note that svn-org@ doesn't have an equivalent @s.a.o list, and that, as
> mentioned upthread, the post-migration (from tigris.org to apache.org)
> mboxes may be in a different order than the official ones, and shouldn't
> be "deduplicated".
>
The mboxes will be preserved but I don't plan to make them available for
download (since they are not available from lists.a.o or mail-archives.a.o).
> You indicate a desire to maintain URLs. Do you have some ideas on that?
>
> Each individual message .shtml file contains the message-id in
> a comment. We can extract the comments and build a redirector around
> them. (By the way, this is basically the same exercise that Infra must
> have solved back when Sebb received that CSV file from the lists.a.o
> vendor, so there may be an opportunity for code reuse.) Of course, the
> full rsync likely has the same info available less scrapily.
>
> Or, as mentioned above, the .shtml files could just be preserved
> statically (plus or minus an appropriate message in the list of years on
> the /${listname}/ page). In fact, I'm having trouble coming up with
> a reason _not_ to serve a static snapshot of the pages, even if we do
> build a redirector.
>
No redirector as of now, only the static [s]html pages.
I will need some help from root to:
1. Install a web server. nginx? (just kidding)
2. Setup httpd.conf
3. Configure a DocumentRoot where I can put the files. Doesn't seem right
to store them in /home
Kind regards
Daniel Sahlberg
Re: svn.haxx.se is going away
Posted by Greg Stein <gs...@gmail.com>.
On Fri, Dec 25, 2020 at 11:17 AM Daniel Shahaf <d....@daniel.shahaf.name>
wrote:
>...
> > I'll figure out a way to have the mboxes downloadable. If I understand
> > Google's documentation of robots.txt they don't care about robots.txt if
> a
> > specific URL is linked from somewhere indexable, they will index it
> anyway.
> > Maybe just make one big tarball of everything?
>
> One big tarball would be wasteful to consume (would have to download
> everything) and to produce (would need to, basically, «cp everything.tgz
> tmp.tgz; tar -zcf - $new >> tmp.tgz; mv tmp.tgz everything.tgz», and you
> can
> see that's O(#everything) rather than O(appended stuff)). Would rather
> avoid
> it if possible.
>
> Not sure what to do about robots. I suppose we could set <link
> rel="canonical"> in the HTTP headers when serving the rfc822 files (example
> in <https://en.wikipedia.org/wiki/Canonical_link_element#HTTP>)?
>
I thought robots.txt can exclude subdirs. So just cut off (say)
svn-haxx.apache.org/mbox/
I'm not too worried about Google crawling the mboxes, as they'll likely do
it just once and never again (by keeping the etag and/or mtime).
>...
> > I couldn't figure out puppet, the links was 404 for me. I've created a
> > request in Jira and I hope someone will take a look:
> > https://issues.apache.org/jira/browse/INFRA-21230
>
> I think the github repository is restricted to Apache committers only, so
> you'll need to enter your github username on id.apache.org in order to get
> access to that URL. If you don't have a github account, there ought to be
> a mirror of the repository on *.apache.org somewhere (at least, if Infra's
> following the same policy PMCs do).
>
Correct: committers only. And only after linking accounts via
https://gitbox.apache.org/setup/ as Nathan noted (and we forgot to mention
to DSahlberg).
If you do not have a GitHub account, or do not want one (say, because you
don't want to accept their T&Cs), then you can use the repository via
gitbox.apache.org (ask on Slack for the link; I prefer not to post it here).
Cheers,
-g
Re: svn.haxx.se is going away
Posted by Nathan Hartman <ha...@gmail.com>.
On Fri, Dec 25, 2020 at 12:17 PM Daniel Shahaf <d....@daniel.shahaf.name>
wrote:
> Daniel Sahlberg wrote on Thu, Dec 24, 2020 at 20:38:17 +0100:
>
> > I couldn't figure out puppet, the links was 404 for me. I've created a
> > request in Jira and I hope someone will take a look:
> > https://issues.apache.org/jira/browse/INFRA-21230
>
> I think the github repository is restricted to Apache committers only, so
> you'll need to enter your github username on id.apache.org in order to get
> access to that URL.
And (if you're going the GitHub route) setup 2 factor authentication. I
think it won't work without that. See:
https://gitbox.apache.org/setup/
Cheers,
Nathan
Re: svn.haxx.se is going away
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Daniel Shahaf wrote on Wed, 20 Jan 2021 15:56 +00:00:
> I see you used CN=svn-haxx.apache.org on the certificate.
>
> Please run this by Infra. It's conceivable that having a *.apache.org
> site that _doesn't_ use the wildcard cert might impact the wildcard's
> reputation in some way (e.g., break certificate pinning rules in
> plugins such as HTTPS Everywhere).
Also, I kinda wonder what trademarks@ would think about the current
setup, since there's this old policy:
https://blogs.apache.org/foundation/entry/if_it_s_not_at "If it's not at apache.org, it's not from the Apache Software Foundation!" (2009)
In particular, I wonder whether they'd prefer that svn.haxx.se redirect
(as in, HTTP 301 responses) to some *.apache.org name.
Thanks for making this happen! These are good problems to have :-)
Daniel
Re: svn.haxx.se is going away
Posted by Daniel Sahlberg <da...@gmail.com>.
Den tors 21 jan. 2021 17:18Daniel Shahaf <d....@daniel.shahaf.name> skrev:
> Daniel Sahlberg wrote on Thu, 21 Jan 2021 07:37 +00:00:
> > Also modified the server setup to redirect any requests for
> > http://svn-haxx.apache.org to https://svn.haxx.se. Only problem now is
> > if someone tries to access https://svn-haxx.apache.org, it will give a
> > certificate warning. I don't really see how we can avoid it without
> > having svn-haxx.a.o in the certificate. (We can't redirect the
> > svn-haxx.a.o DNS entry to another box since the whole purpose of that
> > entry is to be a CNAME target). Anyhow, nobody should be browsing that
> > URL anyway, it shouldn't exist anywhere except in a few mails in dev@.
> >
>
> I'm not concerned about people using unofficial URLs.
>
Me neither. Are you happy with the setup as it is now?
> > > Second, I'm not happy about setting the address to dev@, for several
> > > reasons. One, it's not development-related traffic. Two, IIRC in
> Let's
> > > Encrypt the email address given is the "owner's" address, so if LE ever
> > > need to contact the PMC for whatever reason, automated or otherwise,
> > > that's the address they'd use. Such traffic should go to private@.
> > >
> > > I'm aware you aren't on that list, Daniel. We'll just have to loop
> > > you in on relevant threads. That would mirror the ACL configuration
> > > (for /repos/private/pmc/subversion/machines, and, IIRC, Infra's puppet
> > > repos too).
> >
> > Switched to private@
>
> Thanks. Nothing received so far (just like nothing was received on dev@
> previously).
>
> Cheers,
>
> Daniel
>
Re: svn.haxx.se is going away
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Daniel Sahlberg wrote on Thu, 21 Jan 2021 07:37 +00:00:
> Also modified the server setup to redirect any requests for
> http://svn-haxx.apache.org to https://svn.haxx.se. Only problem now is
> if someone tries to access https://svn-haxx.apache.org, it will give a
> certificate warning. I don't really see how we can avoid it without
> having svn-haxx.a.o in the certificate. (We can't redirect the
> svn-haxx.a.o DNS entry to another box since the whole purpose of that
> entry is to be a CNAME target). Anyhow, nobody should be browsing that
> URL anyway, it shouldn't exist anywhere except in a few mails in dev@.
>
I'm not concerned about people using unofficial URLs.
> > Second, I'm not happy about setting the address to dev@, for several
> > reasons. One, it's not development-related traffic. Two, IIRC in Let's
> > Encrypt the email address given is the "owner's" address, so if LE ever
> > need to contact the PMC for whatever reason, automated or otherwise,
> > that's the address they'd use. Such traffic should go to private@.
> >
> > I'm aware you aren't on that list, Daniel. We'll just have to loop
> > you in on relevant threads. That would mirror the ACL configuration
> > (for /repos/private/pmc/subversion/machines, and, IIRC, Infra's puppet
> > repos too).
>
> Switched to private@
Thanks. Nothing received so far (just like nothing was received on dev@ previously).
Cheers,
Daniel
Re: svn.haxx.se is going away
Posted by Daniel Sahlberg <da...@gmail.com>.
Den ons 20 jan. 2021 kl 16:57 skrev Daniel Shahaf <d....@daniel.shahaf.name>:
> Daniel Sahlberg wrote on Wed, 20 Jan 2021 07:12 +00:00:
> > Den ons 20 jan. 2021 kl 00:16 skrev Nathan Hartman <
> hartman.nathan@gmail.com>:
> > > On Mon, Jan 18, 2021 at 4:17 AM Daniel Sahlberg
> > > <da...@gmail.com> wrote:
> > > > * No SSL at the moment. I suggest to install certbot and a Let's
> encrypt certificate. (Should renewal notices go to dev@?)
> > >
> > > Is there any sensitive information in them? If yes, private@, if not,
> > > dev@ should be fine.
> >
> > Don't think there should be anything sensitive, worst case is probably
> > "Your site's certificate has not been updated and is now EOL, you need
> > to update". I've put it to dev@ and we might reconsider later on.
>
> I see you used CN=svn-haxx.apache.org on the certificate.
>
> Please run this by Infra. It's conceivable that having a *.apache.org
> site that _doesn't_ use the wildcard cert might impact the wildcard's
> reputation in some way (e.g., break certificate pinning rules in
> plugins such as HTTPS Everywhere).
>
> [We aren't going to get a copy of the wildcard cert on a PMC VM, but
> Infra might do an SSL-terminating reverse proxy for us from a box
> they control.]
>
I removed svn-haxx.a.o from the certificate.
Also modified the server setup to redirect any requests for
http://svn-haxx.apache.org to https://svn.haxx.se. Only problem now is if
someone tries to access https://svn-haxx.apache.org, it will give a
certificate warning. I don't really see how we can avoid it without having
svn-haxx.a.o in the certificate. (We can't redirect the svn-haxx.a.o DNS
entry to another box since the whole purpose of that entry is to be a CNAME
target). Anyhow, nobody should be browsing that URL anyway, it shouldn't
exist anywhere except in a few mails in dev@.
Second, I'm not happy about setting the address to dev@, for several
> reasons. One, it's not development-related traffic. Two, IIRC in Let's
> Encrypt the email address given is the "owner's" address, so if LE ever
> need to contact the PMC for whatever reason, automated or otherwise,
> that's the address they'd use. Such traffic should go to private@.
>
> I'm aware you aren't on that list, Daniel. We'll just have to loop
> you in on relevant threads. That would mirror the ACL configuration
> (for /repos/private/pmc/subversion/machines, and, IIRC, Infra's puppet
> repos too).
>
Switched to private@
Kind regards,
Daniel
Re: svn.haxx.se is going away
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Daniel Sahlberg wrote on Wed, 20 Jan 2021 07:12 +00:00:
> Den ons 20 jan. 2021 kl 00:16 skrev Nathan Hartman <ha...@gmail.com>:
> > On Mon, Jan 18, 2021 at 4:17 AM Daniel Sahlberg
> > <da...@gmail.com> wrote:
> > > * No SSL at the moment. I suggest to install certbot and a Let's encrypt certificate. (Should renewal notices go to dev@?)
> >
> > Is there any sensitive information in them? If yes, private@, if not,
> > dev@ should be fine.
>
> Don't think there should be anything sensitive, worst case is probably
> "Your site's certificate has not been updated and is now EOL, you need
> to update". I've put it to dev@ and we might reconsider later on.
I see you used CN=svn-haxx.apache.org on the certificate.
Please run this by Infra. It's conceivable that having a *.apache.org
site that _doesn't_ use the wildcard cert might impact the wildcard's
reputation in some way (e.g., break certificate pinning rules in
plugins such as HTTPS Everywhere).
[We aren't going to get a copy of the wildcard cert on a PMC VM, but
Infra might do an SSL-terminating reverse proxy for us from a box
they control.]
Second, I'm not happy about setting the address to dev@, for several
reasons. One, it's not development-related traffic. Two, IIRC in Let's
Encrypt the email address given is the "owner's" address, so if LE ever
need to contact the PMC for whatever reason, automated or otherwise,
that's the address they'd use. Such traffic should go to private@.
I'm aware you aren't on that list, Daniel. We'll just have to loop
you in on relevant threads. That would mirror the ACL configuration
(for /repos/private/pmc/subversion/machines, and, IIRC, Infra's puppet
repos too).
Cheers,
Daniel
Re: svn.haxx.se is going away
Posted by Daniel Sahlberg <da...@gmail.com>.
Den ons 20 jan. 2021 kl 08:12 skrev Daniel Sahlberg <
daniel.l.sahlberg@gmail.com>:
[...]
> > * I will give it a few days for feedback, if nothing unexpected I will
>> ask Daniel Stenberg to switch the DNS to a CNAME.
>>
>
> I've asked for the DNS update now.
>
The DNS is updated and should have propagated by now.
> > * After the CNAME is installed (and propaged through the DNS system), I
>> will make a final rsync and update the site with any changes (in particular
>> all messages that has arrived since the last sync).
>>
>
I've made the final rsync and updated the site with any messages that has
arrived since december (just in case someone linked them from somewhere).
> > * mboxes are not yet published. I will add this after the final rsync.
>>
>
I've added links to each index page.
Kind regards,
Daniel
Re: svn.haxx.se is going away
Posted by Daniel Sahlberg <da...@gmail.com>.
Den ons 20 jan. 2021 kl 00:16 skrev Nathan Hartman <hartman.nathan@gmail.com
>:
> On Mon, Jan 18, 2021 at 4:17 AM Daniel Sahlberg
> <da...@gmail.com> wrote:
> >
> > The site is now mirrored at http://svn-haxx.apache.org/ (it is also
> listening to http://svn.haxx.se, but the DNS is not yet updated). I
> encourage everyone to take a look and provide feedback.
>
> Thank you so much for driving this and getting it done!!
>
> Feedback -- it looks great. Just a couple of minor nits:
>
> On the main page, the first sentence ("This site was an unofficial
> Subversion related...") contains "archive archived"; it should be just
> "archive".
>
> Also the last sentence ("There is a link to the official archive on
> each page") should have a period.
>
Fixed, thanks!
> > * No SSL at the moment. I suggest to install certbot and a Let's encrypt
> certificate. (Should renewal notices go to dev@?)
>
> Is there any sensitive information in them? If yes, private@, if not,
> dev@ should be fine.
>
Don't think there should be anything sensitive, worst case is probably
"Your site's certificate has not been updated and is now EOL, you need to
update". I've put it to dev@ and we might reconsider later on.
> > * I will give it a few days for feedback, if nothing unexpected I will
> ask Daniel Stenberg to switch the DNS to a CNAME.
>
I've asked for the DNS update now.
> > * After the CNAME is installed (and propaged through the DNS system), I
> will make a final rsync and update the site with any changes (in particular
> all messages that has arrived since the last sync).
> > * mboxes are not yet published. I will add this after the final rsync.
>
> Once again, thank you for driving this.
>
> Nathan
>
Re: svn.haxx.se is going away
Posted by Daniel Sahlberg <da...@gmail.com>.
Den ons 20 jan. 2021 kl 08:54 skrev Andrew Marlow <ma...@gmail.com>:
> Hello everyone,
>
> It looks like a great job has been done on this but there is one little
> wrinkle I have found still. [svn.haxx.se] in the menu bar relocates,
> which is good but individual emails from the archive have that link at the
> bottom of the page still pointing to the obsolete site. Should those links
> also point to https://svn-haxx.apache.org while displaying [svn.haxx.se] ?
>
I believe this might be a problem if you were browsing the site using the
(unofficial - and for technical purposes only) url
https://svn-haxx.apache.org. The links should point to https://svn.haxx.se
which is still the official address.
In the meantime Daniel Stenberg has updated the DNS and the url
https://svn.haxx.se is now pointing at apache hardware (pending DNS
propagation delays). Can you check again on https://svn.haxx.se (possibly
waiting for up to an hour to clear DNS caches, you should see on the front
page if you are looking at the correct server).
Kind regards,
Daniel Sahlberg
Re: svn.haxx.se is going away
Posted by Andrew Marlow <ma...@gmail.com>.
Hello everyone,
It looks like a great job has been done on this but there is one little
wrinkle I have found still. [svn.haxx.se] in the menu bar relocates, which
is good but individual emails from the archive have that link at the bottom
of the page still pointing to the obsolete site. Should those links also
point to https://svn-haxx.apache.org while displaying [svn.haxx.se] ?
On Tue, 19 Jan 2021 at 23:16, Nathan Hartman <ha...@gmail.com>
wrote:
> On Mon, Jan 18, 2021 at 4:17 AM Daniel Sahlberg
> <da...@gmail.com> wrote:
> >
> > The site is now mirrored at http://svn-haxx.apache.org/ (it is also
> listening to http://svn.haxx.se, but the DNS is not yet updated). I
> encourage everyone to take a look and provide feedback.
>
> Thank you so much for driving this and getting it done!!
>
> Feedback -- it looks great. Just a couple of minor nits:
>
> On the main page, the first sentence ("This site was an unofficial
> Subversion related...") contains "archive archived"; it should be just
> "archive".
>
> Also the last sentence ("There is a link to the official archive on
> each page") should have a period.
>
> Other than that, everything looks good to me. I didn't find any other
> issues on any of the pages. The site works (yay!)...
>
> > * No SSL at the moment. I suggest to install certbot and a Let's encrypt
> certificate. (Should renewal notices go to dev@?)
>
> Is there any sensitive information in them? If yes, private@, if not,
> dev@ should be fine.
>
> > * I will give it a few days for feedback, if nothing unexpected I will
> ask Daniel Stenberg to switch the DNS to a CNAME.
> > * After the CNAME is installed (and propaged through the DNS system), I
> will make a final rsync and update the site with any changes (in particular
> all messages that has arrived since the last sync).
> > * mboxes are not yet published. I will add this after the final rsync.
>
> Once again, thank you for driving this.
>
> Nathan
>
--
Regards,
Andrew Marlow
http://www.andrewpetermarlow.co.uk
Re: svn.haxx.se is going away
Posted by Nathan Hartman <ha...@gmail.com>.
On Mon, Jan 18, 2021 at 4:17 AM Daniel Sahlberg
<da...@gmail.com> wrote:
>
> The site is now mirrored at http://svn-haxx.apache.org/ (it is also listening to http://svn.haxx.se, but the DNS is not yet updated). I encourage everyone to take a look and provide feedback.
Thank you so much for driving this and getting it done!!
Feedback -- it looks great. Just a couple of minor nits:
On the main page, the first sentence ("This site was an unofficial
Subversion related...") contains "archive archived"; it should be just
"archive".
Also the last sentence ("There is a link to the official archive on
each page") should have a period.
Other than that, everything looks good to me. I didn't find any other
issues on any of the pages. The site works (yay!)...
> * No SSL at the moment. I suggest to install certbot and a Let's encrypt certificate. (Should renewal notices go to dev@?)
Is there any sensitive information in them? If yes, private@, if not,
dev@ should be fine.
> * I will give it a few days for feedback, if nothing unexpected I will ask Daniel Stenberg to switch the DNS to a CNAME.
> * After the CNAME is installed (and propaged through the DNS system), I will make a final rsync and update the site with any changes (in particular all messages that has arrived since the last sync).
> * mboxes are not yet published. I will add this after the final rsync.
Once again, thank you for driving this.
Nathan
Re: svn.haxx.se is going away
Posted by Daniel Sahlberg <da...@gmail.com>.
The site is now mirrored at http://svn-haxx.apache.org/ (it is also
listening to http://svn.haxx.se, but the DNS is not yet updated). I
encourage everyone to take a look and provide feedback.
Points to note:
* This will be a static mirror from now on.
* No SSL at the moment. I suggest to install certbot and a Let's encrypt
certificate. (Should renewal notices go to dev@?)
* I will give it a few days for feedback, if nothing unexpected I will ask
Daniel Stenberg to switch the DNS to a CNAME.
* After the CNAME is installed (and propaged through the DNS system), I
will make a final rsync and update the site with any changes (in particular
all messages that has arrived since the last sync).
* mboxes are not yet published. I will add this after the final rsync.
/Daniel Sahlberg
Den tors 31 dec. 2020 kl 12:44 skrev Daniel Sahlberg <
daniel.l.sahlberg@gmail.com>:
> Infra replied (in JIRA) that they "doesn't typically get involved in the
> management of project VMs. Is there no one on your project who is able to
> maintain the VM or install software?". I could do it but I don't have root.
> I guess that's a PMC decision not discuessed here.
>
> Let me know if there is anything further I can do.
>
>>
> /Daniel Sahlberg
>
Re: svn.haxx.se is going away
Posted by Daniel Sahlberg <da...@gmail.com>.
Infra replied (in JIRA) that they "doesn't typically get involved in the
management of project VMs. Is there no one on your project who is able to
maintain the VM or install software?". I could do it but I don't have root.
I guess that's a PMC decision not discuessed here.
Let me know if there is anything further I can do.
>
/Daniel Sahlberg
Re: svn.haxx.se is going away
Posted by Daniel Sahlberg <da...@gmail.com>.
Common reply to Daniel Shahaf, Nathan & Greg.
As for mboxes, I'll look at how to make them accessible and take your
suggestions into consideration. Probably just a simple index file listing
all the mboxes (just like today).
I got the gitbox/github integration working and I managed to clone the
repo, but I don't have enough knowledge about puppet to really figure out
the proper change. Thanks for your help!
Den fre 25 dec. 2020 kl 18:17 skrev Daniel Shahaf <d....@daniel.shahaf.name>:
> Post a list of packages you'd like installed?
>
Should be enough with the apache2 package (plus dependencies).
I use the following config on my dev machine. It probably should be
duplicated for svn-haxx.apache.org. The Document Root could/should be
changed to have the site outside of my ~.
<VirtualHost *:80>
ServerName svn.haxx.se
ServerAdmin ????
DocumentRoot /home/dsahlberg/svnhaxx
CustomLog logs/access.log combined
<Directory "/home/dsahlberg/svnhaxx">
Options Indexes MultiViews FollowSymLinks
AllowOverride All
Require all granted
</Directory>
</VirtualHost>
Kind regards
Daniel Sahlberg
Re: svn.haxx.se is going away
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Daniel Sahlberg wrote on Thu, Dec 24, 2020 at 20:38:17 +0100:
> Den tis 22 dec. 2020 kl 02:08 skrev Greg Stein <gs...@gmail.com>:
>
> > On Mon, Dec 21, 2020 at 4:03 AM Daniel Shahaf <d....@daniel.shahaf.name>
> > wrote:
> >
> >> Daniel Sahlberg wrote on Mon, 21 Dec 2020 08:55 +0100:
> >> > Den fre 27 nov. 2020 kl 19:26 skrev Daniel Shahaf <
> >> d.s@daniel.shahaf.name>:
> >> >
> >> > > Sounds good. Nathan, Daniel Sahlberg — could you work with Infra on
> >> > > getting the data over to ASF hardware?
> >> >
> >> > I have been given access to svn-qavm and uploaded a tarball of the
> >> website
> >> > (including mboxes). I'm a bit reluctant to unpack it since it takes
> >> almost
> >> > 7GB, and there is only 14 GB disk space remaining. Is it ok to unpack or
> >> > should we ask Infra for more disk space?
> >>
> >> I vote to ask for more disk space, especially considering that some
> >> percentage is reserved for uid=0's use.
> >>
> >
> > DSahlberg hit up Infra on #asfinfra on the-asf.slack.com, and asked for
> > more space. That's been provisioned now.
> >
>
> I've unpacked in /home/dsahlberg/svnhaxx
>
>
> > >...
> >
> >> > The mboxes will be preserved but I don't plan to make them available for
> >> > download (since they are not available from lists.a.o or
> >> mail-archives.a.o).
> >>
> >> Please do make them available for download. Being able to download the
> >> raw data is useful for both backup and perusal purposes, and I doubt
> >> the bandwidth requirements would be a problem. (Might want
> >> a robots.txt entry, though?)
> >>
> >
> > Bandwidth should not be a problem for the mboxes, but yes: a robots.txt
> > would be nice. I think search engines spidering the static email pages
> > might be useful to the community, but the spiders really shouldn't need/use
> > the mboxes.
> >
>
> I'll figure out a way to have the mboxes downloadable. If I understand
> Google's documentation of robots.txt they don't care about robots.txt if a
> specific URL is linked from somewhere indexable, they will index it anyway.
> Maybe just make one big tarball of everything?
One big tarball would be wasteful to consume (would have to download
everything) and to produce (would need to, basically, «cp everything.tgz
tmp.tgz; tar -zcf - $new >> tmp.tgz; mv tmp.tgz everything.tgz», and you can
see that's O(#everything) rather than O(appended stuff)). Would rather avoid
it if possible.
Not sure what to do about robots. I suppose we could set <link
rel="canonical"> in the HTTP headers when serving the rfc822 files (example
in <https://en.wikipedia.org/wiki/Canonical_link_element#HTTP>)?
> > I think the first thing is to get httpd up and running with the desired
> > configuration. Then step two will be to memorialize that into puppet. Infra
> > can assist with the latter. I saw on Slack that Humbedooh gave you a link
> > to explore.
> >
>
> Since I havn't got root, I can't get any further to install httpd on my own.
Post a list of packages you'd like installed?
> I couldn't figure out puppet, the links was 404 for me. I've created a
> request in Jira and I hope someone will take a look:
> https://issues.apache.org/jira/browse/INFRA-21230
I think the github repository is restricted to Apache committers only, so
you'll need to enter your github username on id.apache.org in order to get
access to that URL. If you don't have a github account, there ought to be
a mirror of the repository on *.apache.org somewhere (at least, if Infra's
following the same policy PMCs do).
Cheers,
Daniel
Re: svn.haxx.se is going away
Posted by Daniel Sahlberg <da...@gmail.com>.
Den tis 22 dec. 2020 kl 02:08 skrev Greg Stein <gs...@gmail.com>:
> On Mon, Dec 21, 2020 at 4:03 AM Daniel Shahaf <d....@daniel.shahaf.name>
> wrote:
>
>> Daniel Sahlberg wrote on Mon, 21 Dec 2020 08:55 +0100:
>> > Den fre 27 nov. 2020 kl 19:26 skrev Daniel Shahaf <
>> d.s@daniel.shahaf.name>:
>> >
>> > > Sounds good. Nathan, Daniel Sahlberg — could you work with Infra on
>> > > getting the data over to ASF hardware?
>> >
>> > I have been given access to svn-qavm and uploaded a tarball of the
>> website
>> > (including mboxes). I'm a bit reluctant to unpack it since it takes
>> almost
>> > 7GB, and there is only 14 GB disk space remaining. Is it ok to unpack or
>> > should we ask Infra for more disk space?
>>
>> I vote to ask for more disk space, especially considering that some
>> percentage is reserved for uid=0's use.
>>
>
> DSahlberg hit up Infra on #asfinfra on the-asf.slack.com, and asked for
> more space. That's been provisioned now.
>
I've unpacked in /home/dsahlberg/svnhaxx
> >...
>
>> > The mboxes will be preserved but I don't plan to make them available for
>> > download (since they are not available from lists.a.o or
>> mail-archives.a.o).
>>
>> Please do make them available for download. Being able to download the
>> raw data is useful for both backup and perusal purposes, and I doubt
>> the bandwidth requirements would be a problem. (Might want
>> a robots.txt entry, though?)
>>
>
> Bandwidth should not be a problem for the mboxes, but yes: a robots.txt
> would be nice. I think search engines spidering the static email pages
> might be useful to the community, but the spiders really shouldn't need/use
> the mboxes.
>
I'll figure out a way to have the mboxes downloadable. If I understand
Google's documentation of robots.txt they don't care about robots.txt if a
specific URL is linked from somewhere indexable, they will index it anyway.
Maybe just make one big tarball of everything?
> I think the first thing is to get httpd up and running with the desired
> configuration. Then step two will be to memorialize that into puppet. Infra
> can assist with the latter. I saw on Slack that Humbedooh gave you a link
> to explore.
>
Since I havn't got root, I can't get any further to install httpd on my own.
I couldn't figure out puppet, the links was 404 for me. I've created a
request in Jira and I hope someone will take a look:
https://issues.apache.org/jira/browse/INFRA-21230
Kind regards,
Daniel
Re: svn.haxx.se is going away
Posted by Greg Stein <gs...@gmail.com>.
On Mon, Dec 21, 2020 at 4:03 AM Daniel Shahaf <d....@daniel.shahaf.name>
wrote:
> Daniel Sahlberg wrote on Mon, 21 Dec 2020 08:55 +0100:
> > Den fre 27 nov. 2020 kl 19:26 skrev Daniel Shahaf <
> d.s@daniel.shahaf.name>:
> >
> > > Sounds good. Nathan, Daniel Sahlberg — could you work with Infra on
> > > getting the data over to ASF hardware?
> >
> > I have been given access to svn-qavm and uploaded a tarball of the
> website
> > (including mboxes). I'm a bit reluctant to unpack it since it takes
> almost
> > 7GB, and there is only 14 GB disk space remaining. Is it ok to unpack or
> > should we ask Infra for more disk space?
>
> I vote to ask for more disk space, especially considering that some
> percentage is reserved for uid=0's use.
>
DSahlberg hit up Infra on #asfinfra on the-asf.slack.com, and asked for
more space. That's been provisioned now.
>...
> > The mboxes will be preserved but I don't plan to make them available for
> > download (since they are not available from lists.a.o or
> mail-archives.a.o).
>
> Please do make them available for download. Being able to download the
> raw data is useful for both backup and perusal purposes, and I doubt
> the bandwidth requirements would be a problem. (Might want
> a robots.txt entry, though?)
>
Bandwidth should not be a problem for the mboxes, but yes: a robots.txt
would be nice. I think search engines spidering the static email pages
might be useful to the community, but the spiders really shouldn't need/use
the mboxes.
Regarding the behaviour of the existing archives, see
> <https://mail-archives.apache.org/mod_mbox/subversion-dev/202012.mbox>
> (which used to also be available via
> https://subversion.apache.org/mail/, but nowadays that just redirects
> to a landing page ☹). I don't know whether lists.a.o has equivalent
> functionality, but then again, lists.a.o has had vendor lock-in baked
> into it from day one, so a lack of a "download raw rfc822 data" feature
> might simply be another form of that.
>
I don't know if our vendor for lists.a.o plans to do an mbox download. I
doubt they retain the data in that format. The Foundation has "all the
data", of course, going back to the mid-90s. An mbox download service might
be interesting, once we decommission the mod_mbox services.
>...
> > 1. Install a web server. nginx? (just kidding)
>
> Apache HTTP Server would probably be a better choice since more dev@svn
> and Infra people are familiar with it, but it's a fair question to ask.
> (Cf. INFRA-7524)
>
Infra has no position on that. Feel free to use nginx 😁 ... but DShahaf is
correct: local support will be higher with apache httpd.
> 2. Setup httpd.conf
> > 3. Configure a DocumentRoot where I can put the files. Doesn't seem right
> > to store them in /home
>
> Hmm. These things should all be done via puppet. I'm not sure what's
> best practice nowadays regarding writing puppet PRs and testing them,
> though.
I think the first thing is to get httpd up and running with the desired
configuration. Then step two will be to memorialize that into puppet. Infra
can assist with the latter. I saw on Slack that Humbedooh gave you a link
to explore.
Cheers,
-g
Re: svn.haxx.se is going away
Posted by Nathan Hartman <ha...@gmail.com>.
On Mon, Dec 21, 2020 at 5:04 AM Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> Daniel Sahlberg wrote on Mon, 21 Dec 2020 08:55 +0100:
> > I have been given access to svn-qavm and uploaded a tarball of the website
> > (including mboxes). I'm a bit reluctant to unpack it since it takes almost
> > 7GB, and there is only 14 GB disk space remaining. Is it ok to unpack or
> > should we ask Infra for more disk space?
> >
>
> I vote to ask for more disk space, especially considering that some
> percentage is reserved for uid=0's use.
I sent users@infra an email asking for more disk space.
Nathan
Re: svn.haxx.se is going away
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Daniel Sahlberg wrote on Mon, 21 Dec 2020 08:55 +0100:
> Den fre 27 nov. 2020 kl 19:26 skrev Daniel Shahaf <d....@daniel.shahaf.name>:
>
> > Sounds good. Nathan, Daniel Sahlberg — could you work with Infra on
> > getting the data over to ASF hardware?
> >
>
> I have been given access to svn-qavm and uploaded a tarball of the website
> (including mboxes). I'm a bit reluctant to unpack it since it takes almost
> 7GB, and there is only 14 GB disk space remaining. Is it ok to unpack or
> should we ask Infra for more disk space?
>
I vote to ask for more disk space, especially considering that some
percentage is reserved for uid=0's use.
> > Note that svn-org@ doesn't have an equivalent @s.a.o list, and that, as
> > mentioned upthread, the post-migration (from tigris.org to apache.org)
> > mboxes may be in a different order than the official ones, and shouldn't
> > be "deduplicated".
> >
>
> The mboxes will be preserved but I don't plan to make them available for
> download (since they are not available from lists.a.o or mail-archives.a.o).
>
Please do make them available for download. Being able to download the
raw data is useful for both backup and perusal purposes, and I doubt
the bandwidth requirements would be a problem. (Might want
a robots.txt entry, though?)
Regarding the behaviour of the existing archives, see
<https://mail-archives.apache.org/mod_mbox/subversion-dev/202012.mbox>
(which used to also be available via
https://subversion.apache.org/mail/, but nowadays that just redirects
to a landing page ☹). I don't know whether lists.a.o has equivalent
functionality, but then again, lists.a.o has had vendor lock-in baked
into it from day one, so a lack of a "download raw rfc822 data" feature
might simply be another form of that.
The mod_mbox product is owned by dev@httpd.
> > You indicate a desire to maintain URLs. Do you have some ideas on that?
> >
> > Each individual message .shtml file contains the message-id in
> > a comment. We can extract the comments and build a redirector around
> > them. (By the way, this is basically the same exercise that Infra must
> > have solved back when Sebb received that CSV file from the lists.a.o
> > vendor, so there may be an opportunity for code reuse.) Of course, the
> > full rsync likely has the same info available less scrapily.
> >
> > Or, as mentioned above, the .shtml files could just be preserved
> > statically (plus or minus an appropriate message in the list of years on
> > the /${listname}/ page). In fact, I'm having trouble coming up with
> > a reason _not_ to serve a static snapshot of the pages, even if we do
> > build a redirector.
> >
>
> No redirector as of now, only the static [s]html pages.
>
<glass type="half-full">Yay!</glass>
> I will need some help from root to:
Not me, I'm afraid; ENOTIME.
> 1. Install a web server. nginx? (just kidding)
Apache HTTP Server would probably be a better choice since more dev@svn
and Infra people are familiar with it, but it's a fair question to ask.
(Cf. INFRA-7524)
> 2. Setup httpd.conf
> 3. Configure a DocumentRoot where I can put the files. Doesn't seem right
> to store them in /home
Hmm. These things should all be done via puppet. I'm not sure what's
best practice nowadays regarding writing puppet PRs and testing them,
though.
Cheers,
Daniel