You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whimsical.apache.org by sebb <se...@gmail.com> on 2019/04/30 12:12:06 UTC
Moving Whimsy to new host: hermes cronjob updates
There are two cronjobs on hermes that copy data to whimsy:
listmodsubs.sh
- copies data files to whimsy.apache.org:/srv/subscriptions/
whimsy_qmail_ids.sh
- writes data to whimsy-vm4.apache.org:/srv/subscriptions/qmail.ids
It would be nice if there were no need to update these scripts when
the Whimsy VM is updated.
I propose to update whimsy_qmail_ids.sh to use the generic address.
Both scripts would then only send data to the current master server.
For testing the new server, one can just copy over the contents of
/srv/subscriptions.
When DNS is flipped, hermes would start to update the new master.
Does that make sense?
S.
Re: Moving Whimsy to new host: hermes cronjob updates
Posted by Chris Lambertus <cm...@apache.org>.
I would respectfully urge caution here, certainly not because of Dave, but in general: the whimsy server itself, along with the code which runs on it, has far more privileges to extract data from the Foundation than just about anything or anyone other than root@. Some of the data that Whimsy has access to is PII, and should be protected thusly. It may not be appropriate to rsync certain data sets to a personal computer, even a Member's, so please be cognizant of that when downloading data for testing/development purposes.
-Chris
> On Apr 30, 2019, at 9:25 AM, Sam Ruby <ru...@intertwingly.net> wrote:
>
> As you are an ASF member, file an infra JIRA to give you shell access
> to whimsy-vm4. Then you can rsync this or any other directory
> whenever you want. As this is a minor request, you might even be able
> to get this done by asking nicely on hipchat^h^h^h^h^h^h^hslack.
>
> - Sam Ruby
>
> On Tue, Apr 30, 2019 at 11:38 AM Dave Fisher <da...@comcast.net> wrote:
>>
>> Hi -
>>
>> I need /srv/subscriptions to properly debug podling roster page changes.
>>
>> How can I get a one time copy of /srv/subscriptions on my local and how big is it?
>>
>> Regards,
>> Dave
>>
>>> On Apr 30, 2019, at 6:30 AM, sebb <se...@gmail.com> wrote:
>>>
>>> On Tue, 30 Apr 2019 at 13:44, Sam Ruby <ru...@intertwingly.net> wrote:
>>>>
>>>> Just a perspective from the one who set up whimsy vm 1 though 4. When
>>>> I was setting up whimsy 3 and 4, I updated those scripts to send data
>>>> to whimsy n-1 and n. Once the setup was completed, I removed n-1 from
>>>> the instructions.
>>>
>>> Yes, though it's not how listmodsubs.sh is set up currently.
>>> [That can be fixed.]
>>>
>>>> The issue is one of timing. If one copies over the contents of
>>>> various directories, and then later updates the DNS, there may be
>>>> missing updates.
>>>
>>> There is only a single directory involved here: /srv/subscriptions
>>> The scripts are run every hour, and the output replaces whatever was
>>> there before.
>>> So whilst the data might be stale immediately after DNS changeover, it
>>> will soon be updated.
>>> I don't think there is any data that can be permanently lost here.
>>>
>>> However it does occur to me that it's possible that the rsync might
>>> fail to work initially on the new host.
>>> Probably best not to discover that first on DNS changeover...
>>>
>>> So I propose now to ensure listmodsubs.sh uses the same approach as
>>> whimsy_qmail_ids.sh
>>> (and document the reasoning)
>>>
>>>> - Sam Ruby
>>>>
>>>> On Tue, Apr 30, 2019 at 8:12 AM sebb <se...@gmail.com> wrote:
>>>>>
>>>>> There are two cronjobs on hermes that copy data to whimsy:
>>>>>
>>>>> listmodsubs.sh
>>>>> - copies data files to whimsy.apache.org:/srv/subscriptions/
>>>>>
>>>>> whimsy_qmail_ids.sh
>>>>> - writes data to whimsy-vm4.apache.org:/srv/subscriptions/qmail.ids
>>>>>
>>>>> It would be nice if there were no need to update these scripts when
>>>>> the Whimsy VM is updated.
>>>>>
>>>>> I propose to update whimsy_qmail_ids.sh to use the generic address.
>>>>> Both scripts would then only send data to the current master server.
>>>>>
>>>>> For testing the new server, one can just copy over the contents of
>>>>> /srv/subscriptions.
>>>>>
>>>>> When DNS is flipped, hermes would start to update the new master.
>>>>>
>>>>> Does that make sense?
>>>>>
>>>>> S.
>>
Re: Moving Whimsy to new host: hermes cronjob updates
Posted by Sam Ruby <ru...@intertwingly.net>.
As you are an ASF member, file an infra JIRA to give you shell access
to whimsy-vm4. Then you can rsync this or any other directory
whenever you want. As this is a minor request, you might even be able
to get this done by asking nicely on hipchat^h^h^h^h^h^h^hslack.
- Sam Ruby
On Tue, Apr 30, 2019 at 11:38 AM Dave Fisher <da...@comcast.net> wrote:
>
> Hi -
>
> I need /srv/subscriptions to properly debug podling roster page changes.
>
> How can I get a one time copy of /srv/subscriptions on my local and how big is it?
>
> Regards,
> Dave
>
> > On Apr 30, 2019, at 6:30 AM, sebb <se...@gmail.com> wrote:
> >
> > On Tue, 30 Apr 2019 at 13:44, Sam Ruby <ru...@intertwingly.net> wrote:
> >>
> >> Just a perspective from the one who set up whimsy vm 1 though 4. When
> >> I was setting up whimsy 3 and 4, I updated those scripts to send data
> >> to whimsy n-1 and n. Once the setup was completed, I removed n-1 from
> >> the instructions.
> >
> > Yes, though it's not how listmodsubs.sh is set up currently.
> > [That can be fixed.]
> >
> >> The issue is one of timing. If one copies over the contents of
> >> various directories, and then later updates the DNS, there may be
> >> missing updates.
> >
> > There is only a single directory involved here: /srv/subscriptions
> > The scripts are run every hour, and the output replaces whatever was
> > there before.
> > So whilst the data might be stale immediately after DNS changeover, it
> > will soon be updated.
> > I don't think there is any data that can be permanently lost here.
> >
> > However it does occur to me that it's possible that the rsync might
> > fail to work initially on the new host.
> > Probably best not to discover that first on DNS changeover...
> >
> > So I propose now to ensure listmodsubs.sh uses the same approach as
> > whimsy_qmail_ids.sh
> > (and document the reasoning)
> >
> >> - Sam Ruby
> >>
> >> On Tue, Apr 30, 2019 at 8:12 AM sebb <se...@gmail.com> wrote:
> >>>
> >>> There are two cronjobs on hermes that copy data to whimsy:
> >>>
> >>> listmodsubs.sh
> >>> - copies data files to whimsy.apache.org:/srv/subscriptions/
> >>>
> >>> whimsy_qmail_ids.sh
> >>> - writes data to whimsy-vm4.apache.org:/srv/subscriptions/qmail.ids
> >>>
> >>> It would be nice if there were no need to update these scripts when
> >>> the Whimsy VM is updated.
> >>>
> >>> I propose to update whimsy_qmail_ids.sh to use the generic address.
> >>> Both scripts would then only send data to the current master server.
> >>>
> >>> For testing the new server, one can just copy over the contents of
> >>> /srv/subscriptions.
> >>>
> >>> When DNS is flipped, hermes would start to update the new master.
> >>>
> >>> Does that make sense?
> >>>
> >>> S.
>
Re: Moving Whimsy to new host: hermes cronjob updates
Posted by Dave Fisher <da...@comcast.net>.
Hi -
> On Apr 30, 2019, at 8:56 AM, sebb <se...@gmail.com> wrote:
>
> On Tue, 30 Apr 2019 at 16:38, Dave Fisher <da...@comcast.net> wrote:
>>
>> Hi -
>>
>> I need /srv/subscriptions to properly debug podling roster page changes.
>>
>> How can I get a one time copy of /srv/subscriptions on my local and how big is it?
>
> 5.9M, though presumable you don't need all the data.
>
> Maybe there should be some sample data that can be used for testing?
>
> The format is very simple:
>
> --- start ---
>
> /home/apmail/lists/accumulo.apache.org/commits
> archive@mail-archive.com
> ...
>
> /home/apmail/lists/accumulo.apache.org/dev
> ...
> ---- end ---
>
> i.e. a blank line before each path name, followed by 0 or more subscribers.
>
> There are 3 files:
> list-digs, -mods, -subs and an empty list-start file touched when the
> data extraction starts.
I think I need a current list one-time as a shortened one will limit testing in various ways.
The reason is that I’ve seen various pages (not necessarily roster) breaking when subscriptions are NIL. I’d be willing to submit issues in these cases, but having the basics will help.
Regards,
Dave
>
>
>> Regards,
>> Dave
>>
>>> On Apr 30, 2019, at 6:30 AM, sebb <se...@gmail.com> wrote:
>>>
>>> On Tue, 30 Apr 2019 at 13:44, Sam Ruby <ru...@intertwingly.net> wrote:
>>>>
>>>> Just a perspective from the one who set up whimsy vm 1 though 4. When
>>>> I was setting up whimsy 3 and 4, I updated those scripts to send data
>>>> to whimsy n-1 and n. Once the setup was completed, I removed n-1 from
>>>> the instructions.
>>>
>>> Yes, though it's not how listmodsubs.sh is set up currently.
>>> [That can be fixed.]
>>>
>>>> The issue is one of timing. If one copies over the contents of
>>>> various directories, and then later updates the DNS, there may be
>>>> missing updates.
>>>
>>> There is only a single directory involved here: /srv/subscriptions
>>> The scripts are run every hour, and the output replaces whatever was
>>> there before.
>>> So whilst the data might be stale immediately after DNS changeover, it
>>> will soon be updated.
>>> I don't think there is any data that can be permanently lost here.
>>>
>>> However it does occur to me that it's possible that the rsync might
>>> fail to work initially on the new host.
>>> Probably best not to discover that first on DNS changeover...
>>>
>>> So I propose now to ensure listmodsubs.sh uses the same approach as
>>> whimsy_qmail_ids.sh
>>> (and document the reasoning)
>>>
>>>> - Sam Ruby
>>>>
>>>> On Tue, Apr 30, 2019 at 8:12 AM sebb <se...@gmail.com> wrote:
>>>>>
>>>>> There are two cronjobs on hermes that copy data to whimsy:
>>>>>
>>>>> listmodsubs.sh
>>>>> - copies data files to whimsy.apache.org:/srv/subscriptions/
>>>>>
>>>>> whimsy_qmail_ids.sh
>>>>> - writes data to whimsy-vm4.apache.org:/srv/subscriptions/qmail.ids
>>>>>
>>>>> It would be nice if there were no need to update these scripts when
>>>>> the Whimsy VM is updated.
>>>>>
>>>>> I propose to update whimsy_qmail_ids.sh to use the generic address.
>>>>> Both scripts would then only send data to the current master server.
>>>>>
>>>>> For testing the new server, one can just copy over the contents of
>>>>> /srv/subscriptions.
>>>>>
>>>>> When DNS is flipped, hermes would start to update the new master.
>>>>>
>>>>> Does that make sense?
>>>>>
>>>>> S.
>>
Re: Moving Whimsy to new host: hermes cronjob updates
Posted by sebb <se...@gmail.com>.
On Tue, 30 Apr 2019 at 16:38, Dave Fisher <da...@comcast.net> wrote:
>
> Hi -
>
> I need /srv/subscriptions to properly debug podling roster page changes.
>
> How can I get a one time copy of /srv/subscriptions on my local and how big is it?
5.9M, though presumable you don't need all the data.
Maybe there should be some sample data that can be used for testing?
The format is very simple:
--- start ---
/home/apmail/lists/accumulo.apache.org/commits
archive@mail-archive.com
...
/home/apmail/lists/accumulo.apache.org/dev
...
---- end ---
i.e. a blank line before each path name, followed by 0 or more subscribers.
There are 3 files:
list-digs, -mods, -subs and an empty list-start file touched when the
data extraction starts.
> Regards,
> Dave
>
> > On Apr 30, 2019, at 6:30 AM, sebb <se...@gmail.com> wrote:
> >
> > On Tue, 30 Apr 2019 at 13:44, Sam Ruby <ru...@intertwingly.net> wrote:
> >>
> >> Just a perspective from the one who set up whimsy vm 1 though 4. When
> >> I was setting up whimsy 3 and 4, I updated those scripts to send data
> >> to whimsy n-1 and n. Once the setup was completed, I removed n-1 from
> >> the instructions.
> >
> > Yes, though it's not how listmodsubs.sh is set up currently.
> > [That can be fixed.]
> >
> >> The issue is one of timing. If one copies over the contents of
> >> various directories, and then later updates the DNS, there may be
> >> missing updates.
> >
> > There is only a single directory involved here: /srv/subscriptions
> > The scripts are run every hour, and the output replaces whatever was
> > there before.
> > So whilst the data might be stale immediately after DNS changeover, it
> > will soon be updated.
> > I don't think there is any data that can be permanently lost here.
> >
> > However it does occur to me that it's possible that the rsync might
> > fail to work initially on the new host.
> > Probably best not to discover that first on DNS changeover...
> >
> > So I propose now to ensure listmodsubs.sh uses the same approach as
> > whimsy_qmail_ids.sh
> > (and document the reasoning)
> >
> >> - Sam Ruby
> >>
> >> On Tue, Apr 30, 2019 at 8:12 AM sebb <se...@gmail.com> wrote:
> >>>
> >>> There are two cronjobs on hermes that copy data to whimsy:
> >>>
> >>> listmodsubs.sh
> >>> - copies data files to whimsy.apache.org:/srv/subscriptions/
> >>>
> >>> whimsy_qmail_ids.sh
> >>> - writes data to whimsy-vm4.apache.org:/srv/subscriptions/qmail.ids
> >>>
> >>> It would be nice if there were no need to update these scripts when
> >>> the Whimsy VM is updated.
> >>>
> >>> I propose to update whimsy_qmail_ids.sh to use the generic address.
> >>> Both scripts would then only send data to the current master server.
> >>>
> >>> For testing the new server, one can just copy over the contents of
> >>> /srv/subscriptions.
> >>>
> >>> When DNS is flipped, hermes would start to update the new master.
> >>>
> >>> Does that make sense?
> >>>
> >>> S.
>
Re: Moving Whimsy to new host: hermes cronjob updates
Posted by Dave Fisher <da...@comcast.net>.
Hi -
I need /srv/subscriptions to properly debug podling roster page changes.
How can I get a one time copy of /srv/subscriptions on my local and how big is it?
Regards,
Dave
> On Apr 30, 2019, at 6:30 AM, sebb <se...@gmail.com> wrote:
>
> On Tue, 30 Apr 2019 at 13:44, Sam Ruby <ru...@intertwingly.net> wrote:
>>
>> Just a perspective from the one who set up whimsy vm 1 though 4. When
>> I was setting up whimsy 3 and 4, I updated those scripts to send data
>> to whimsy n-1 and n. Once the setup was completed, I removed n-1 from
>> the instructions.
>
> Yes, though it's not how listmodsubs.sh is set up currently.
> [That can be fixed.]
>
>> The issue is one of timing. If one copies over the contents of
>> various directories, and then later updates the DNS, there may be
>> missing updates.
>
> There is only a single directory involved here: /srv/subscriptions
> The scripts are run every hour, and the output replaces whatever was
> there before.
> So whilst the data might be stale immediately after DNS changeover, it
> will soon be updated.
> I don't think there is any data that can be permanently lost here.
>
> However it does occur to me that it's possible that the rsync might
> fail to work initially on the new host.
> Probably best not to discover that first on DNS changeover...
>
> So I propose now to ensure listmodsubs.sh uses the same approach as
> whimsy_qmail_ids.sh
> (and document the reasoning)
>
>> - Sam Ruby
>>
>> On Tue, Apr 30, 2019 at 8:12 AM sebb <se...@gmail.com> wrote:
>>>
>>> There are two cronjobs on hermes that copy data to whimsy:
>>>
>>> listmodsubs.sh
>>> - copies data files to whimsy.apache.org:/srv/subscriptions/
>>>
>>> whimsy_qmail_ids.sh
>>> - writes data to whimsy-vm4.apache.org:/srv/subscriptions/qmail.ids
>>>
>>> It would be nice if there were no need to update these scripts when
>>> the Whimsy VM is updated.
>>>
>>> I propose to update whimsy_qmail_ids.sh to use the generic address.
>>> Both scripts would then only send data to the current master server.
>>>
>>> For testing the new server, one can just copy over the contents of
>>> /srv/subscriptions.
>>>
>>> When DNS is flipped, hermes would start to update the new master.
>>>
>>> Does that make sense?
>>>
>>> S.
Re: Moving Whimsy to new host: hermes cronjob updates
Posted by sebb <se...@gmail.com>.
On Tue, 30 Apr 2019 at 13:44, Sam Ruby <ru...@intertwingly.net> wrote:
>
> Just a perspective from the one who set up whimsy vm 1 though 4. When
> I was setting up whimsy 3 and 4, I updated those scripts to send data
> to whimsy n-1 and n. Once the setup was completed, I removed n-1 from
> the instructions.
Yes, though it's not how listmodsubs.sh is set up currently.
[That can be fixed.]
> The issue is one of timing. If one copies over the contents of
> various directories, and then later updates the DNS, there may be
> missing updates.
There is only a single directory involved here: /srv/subscriptions
The scripts are run every hour, and the output replaces whatever was
there before.
So whilst the data might be stale immediately after DNS changeover, it
will soon be updated.
I don't think there is any data that can be permanently lost here.
However it does occur to me that it's possible that the rsync might
fail to work initially on the new host.
Probably best not to discover that first on DNS changeover...
So I propose now to ensure listmodsubs.sh uses the same approach as
whimsy_qmail_ids.sh
(and document the reasoning)
> - Sam Ruby
>
> On Tue, Apr 30, 2019 at 8:12 AM sebb <se...@gmail.com> wrote:
> >
> > There are two cronjobs on hermes that copy data to whimsy:
> >
> > listmodsubs.sh
> > - copies data files to whimsy.apache.org:/srv/subscriptions/
> >
> > whimsy_qmail_ids.sh
> > - writes data to whimsy-vm4.apache.org:/srv/subscriptions/qmail.ids
> >
> > It would be nice if there were no need to update these scripts when
> > the Whimsy VM is updated.
> >
> > I propose to update whimsy_qmail_ids.sh to use the generic address.
> > Both scripts would then only send data to the current master server.
> >
> > For testing the new server, one can just copy over the contents of
> > /srv/subscriptions.
> >
> > When DNS is flipped, hermes would start to update the new master.
> >
> > Does that make sense?
> >
> > S.
Re: Moving Whimsy to new host: hermes cronjob updates
Posted by Sam Ruby <ru...@intertwingly.net>.
Just a perspective from the one who set up whimsy vm 1 though 4. When
I was setting up whimsy 3 and 4, I updated those scripts to send data
to whimsy n-1 and n. Once the setup was completed, I removed n-1 from
the instructions.
The issue is one of timing. If one copies over the contents of
various directories, and then later updates the DNS, there may be
missing updates.
- Sam Ruby
On Tue, Apr 30, 2019 at 8:12 AM sebb <se...@gmail.com> wrote:
>
> There are two cronjobs on hermes that copy data to whimsy:
>
> listmodsubs.sh
> - copies data files to whimsy.apache.org:/srv/subscriptions/
>
> whimsy_qmail_ids.sh
> - writes data to whimsy-vm4.apache.org:/srv/subscriptions/qmail.ids
>
> It would be nice if there were no need to update these scripts when
> the Whimsy VM is updated.
>
> I propose to update whimsy_qmail_ids.sh to use the generic address.
> Both scripts would then only send data to the current master server.
>
> For testing the new server, one can just copy over the contents of
> /srv/subscriptions.
>
> When DNS is flipped, hermes would start to update the new master.
>
> Does that make sense?
>
> S.