You are viewing a plain text version of this content. The canonical link for it is here.
Posted to repo-maintainers@maven.apache.org by Carlos Sanchez <ca...@apache.org> on 2008/02/20 03:45:48 UTC
New sync process in place
I have finished a new process for Maven 2 repo synchronization, it's
coded in java and uses a csv file for the list of repositories
https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper/src/bin/synchronize/m2-sync/sync.csv
This will make easier to add new repositories.
The java code just executes rsync for each repo as the old script did.
New features:
* in error both this list and the repo owner will be emailed
Jason, are you receiving maven@maven.org emails? we'd need to
subscribe that address to this list, or change the address to
something else
* there's a 5 minutes timeout for each repo, if it doesnt get synced
it's reported as an error and it moves on to the next repo
The code is at https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper
I've also splitted the big synchronization script in different pieces
so we sync the m1 repos less often (they are not updated as much), as
the sync to the ibiblio mirror (takes a long time), and so in case of
errors in one part (eg ibiblio sync) the others continue (eg. m2 sync
to repo1)
The new crontab is at
https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper/src/bin/crontab.txt
Please keep an eye if you make new releases to make sure everything
works fine and shows up in the central repo and the main mirrors
http://repo1.maven.org/maven2/
http://mirrors.ibiblio.org/pub/mirrors/maven2/
http://ftp.cica.es/mirrors/maven2/
Thanks
--
I could give you my word as a Spaniard.
No good. I've known too many Spaniards.
-- The Princess Bride
Re: New sync process in place
Posted by Jason van Zyl <ja...@maven.org>.
Contegix uses Hyperic for all the monitoring of the machines that are
used:
http://www.hyperic.com/
I'm sure Contegix can do anything necessary but I know the Hyperic
folks and if there is anything we need from that side we'll get the
support.
On 20-Feb-08, at 1:41 AM, Brett Porter wrote:
>
> On 20/02/2008, at 6:23 PM, Jason van Zyl wrote:
>
>>
>> On 19-Feb-08, at 8:30 PM, Carlos Sanchez wrote:
>>
>>> the problem with the m2 synchronization scripts is that we need to
>>> add
>>> one per groupId synced, right now there are 80+ groups to be synced
>>> and is a pain to mantain.
>>
>> And so you had to write it in Java why? I just find it totally odd
>> that you ignored the first set of tools written in perl which were
>> perfectly fine, into a set of not so good shell scripts which
>> blobbed every back together, and now into Java. Perl does fine with
>> threads, timeouts, what have you and sysadmin folks have an easier
>> time with it.
>
> Speaking for myself I would find the Java stuff easier to make
> changes on when I needed to (I've been super-tentative about the
> changes I made to the perl scripts in the past).
>
>> Contegix being the important ones. If you had asked them they would
>> have been more then happy to make a decent configuration, and
>> something with better process control, time outs and simple
>> reporting. They whip things like that up all the time and they like
>> doing it and they are the ones around 24/7.
>
> I'm sure Carlos is in the same boat, and it wouldn't have occurred
> to me that they would do that. I think it makes sense if they can.
>
> How about Carlos get in touch with them and explain the two things
> we have, what we need requirements-wise and see what they'd prefer
> to do? I'd say what we essentially want is just for the syncs to be
> monitored, nag people when configuration isn't right, and take
> updates from SVN when we add/change configs, right?
>
> Cheers,
> Brett
>
> --
> Brett Porter
> brett@apache.org
> http://blogs.exist.com/bporter/
>
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Apache Maven
jason at sonatype dot com
----------------------------------------------------------
First, the taking in of scattered particulars under one Idea,
so that everyone understands what is being talked about ... Second,
the separation of the Idea into parts, by dividing it at the joints,
as nature directs, not breaking any limb in half as a bad carver might.
-- Plato, Phaedrus (Notes on the Synthesis of Form by C. Alexander)
Re: New sync process in place
Posted by Jason van Zyl <ja...@maven.org>.
On 20-Feb-08, at 1:41 AM, Brett Porter wrote:
>
> On 20/02/2008, at 6:23 PM, Jason van Zyl wrote:
>
>>
>> On 19-Feb-08, at 8:30 PM, Carlos Sanchez wrote:
>>
>>> the problem with the m2 synchronization scripts is that we need to
>>> add
>>> one per groupId synced, right now there are 80+ groups to be synced
>>> and is a pain to mantain.
>>
>> And so you had to write it in Java why? I just find it totally odd
>> that you ignored the first set of tools written in perl which were
>> perfectly fine, into a set of not so good shell scripts which
>> blobbed every back together, and now into Java. Perl does fine with
>> threads, timeouts, what have you and sysadmin folks have an easier
>> time with it.
>
> Speaking for myself I would find the Java stuff easier to make
> changes on when I needed to (I've been super-tentative about the
> changes I made to the perl scripts in the past).
>
You're a Java programmer 99.9% of the time so that's understandable.
>> Contegix being the important ones. If you had asked them they would
>> have been more then happy to make a decent configuration, and
>> something with better process control, time outs and simple
>> reporting. They whip things like that up all the time and they like
>> doing it and they are the ones around 24/7.
>
> I'm sure Carlos is in the same boat, and it wouldn't have occurred
> to me that they would do that. I think it makes sense if they can.
>
> How about Carlos get in touch with them and explain the two things
> we have, what we need requirements-wise and see what they'd prefer
> to do? I'd say what we essentially want is just for the syncs to be
> monitored, nag people when configuration isn't right, and take
> updates from SVN when we add/change configs, right?
>
We can give them any of this, they are open and willing. I assume you
looked at this stuff and it's fine adding reporting stuff, indexing
stuff, logging, little webapps or whatever, but changing the way the
sync happens or putting big piles of stuff into the repository without
vetting it by a single other person just seems unwise.
> Cheers,
> Brett
>
> --
> Brett Porter
> brett@apache.org
> http://blogs.exist.com/bporter/
>
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Apache Maven
jason at sonatype dot com
----------------------------------------------------------
We know what we are, but know not what we may be.
-- Shakespeare
Re: New sync process in place
Posted by Brett Porter <br...@apache.org>.
On 20/02/2008, at 6:23 PM, Jason van Zyl wrote:
>
> On 19-Feb-08, at 8:30 PM, Carlos Sanchez wrote:
>
>> the problem with the m2 synchronization scripts is that we need to
>> add
>> one per groupId synced, right now there are 80+ groups to be synced
>> and is a pain to mantain.
>
> And so you had to write it in Java why? I just find it totally odd
> that you ignored the first set of tools written in perl which were
> perfectly fine, into a set of not so good shell scripts which
> blobbed every back together, and now into Java. Perl does fine with
> threads, timeouts, what have you and sysadmin folks have an easier
> time with it.
Speaking for myself I would find the Java stuff easier to make changes
on when I needed to (I've been super-tentative about the changes I
made to the perl scripts in the past).
> Contegix being the important ones. If you had asked them they would
> have been more then happy to make a decent configuration, and
> something with better process control, time outs and simple
> reporting. They whip things like that up all the time and they like
> doing it and they are the ones around 24/7.
I'm sure Carlos is in the same boat, and it wouldn't have occurred to
me that they would do that. I think it makes sense if they can.
How about Carlos get in touch with them and explain the two things we
have, what we need requirements-wise and see what they'd prefer to do?
I'd say what we essentially want is just for the syncs to be
monitored, nag people when configuration isn't right, and take updates
from SVN when we add/change configs, right?
Cheers,
Brett
--
Brett Porter
brett@apache.org
http://blogs.exist.com/bporter/
Re: New sync process in place
Posted by Christian Edward Gruber <cg...@israfil.net>.
I don't know - with commons-cli and some other things, it's not
entirely unworkable. It solves the multi-platform issue, etc. Having
said that, unless I need to do threading stuff, I do tend to write sh
scripts. Bottom line, though... it's an NP-complete language -
neither better nor worse at most things except from a style/
convenience perspective.
Christian.
On 21-Feb-08, at 15:44 , Henri Yandell wrote:
> On Wed, Feb 20, 2008 at 9:51 AM, Carlos Sanchez <ca...@apache.org>
> wrote:
>> On Feb 19, 2008 11:23 PM, Jason van Zyl <ja...@maven.org> wrote:
>>>
>>> On 19-Feb-08, at 8:30 PM, Carlos Sanchez wrote:
>>>
>>>> the problem with the m2 synchronization scripts is that we need
>>>> to add
>>>> one per groupId synced, right now there are 80+ groups to be synced
>>>> and is a pain to mantain.
>>>
>>> And so you had to write it in Java why? I just find it totally odd
>>> that you ignored the first set of tools written in perl which were
>>> perfectly fine, into a set of not so good shell scripts which
>>> blobbed
>>> every back together, and now into Java. Perl does fine with threads,
>>> timeouts, what have you and sysadmin folks have an easier time
>>> with it.
>>
>>
>> The Perl tools were done for maven 1 sync and didn't work for m2
>> (sync
>> the whole repo instead of specific groups amongs other problems)
>> The m2 scripts were the fastest way I had to get m2 syncs in place
>> when people started asking for them (there were several weeks when we
>> had m1 sync and no m2 sync)
>> They could have been done in Perl? sure, but it would take me even
>> more time to do it and considering that Maven devs are 99% Java
>> developers I thought people would be more willing to contrib, fix
>> bugs,... in Java
>
> Writing scripts in Java *shudder*. You guys are sick.
>
> I think I'm a Java weenie and then this kind of thing comes along and
> I realize I'm some kind of multiple-personalitied
> Java-weenie/script-kiddie/god-knows-what-else.
>
> Bet they end up being in Ruby in a years time.
>
> Hen
>
Re: New sync process in place
Posted by Henri Yandell <ba...@apache.org>.
On Wed, Feb 20, 2008 at 9:51 AM, Carlos Sanchez <ca...@apache.org> wrote:
> On Feb 19, 2008 11:23 PM, Jason van Zyl <ja...@maven.org> wrote:
> >
> > On 19-Feb-08, at 8:30 PM, Carlos Sanchez wrote:
> >
> > > the problem with the m2 synchronization scripts is that we need to add
> > > one per groupId synced, right now there are 80+ groups to be synced
> > > and is a pain to mantain.
> >
> > And so you had to write it in Java why? I just find it totally odd
> > that you ignored the first set of tools written in perl which were
> > perfectly fine, into a set of not so good shell scripts which blobbed
> > every back together, and now into Java. Perl does fine with threads,
> > timeouts, what have you and sysadmin folks have an easier time with it.
>
>
> The Perl tools were done for maven 1 sync and didn't work for m2 (sync
> the whole repo instead of specific groups amongs other problems)
> The m2 scripts were the fastest way I had to get m2 syncs in place
> when people started asking for them (there were several weeks when we
> had m1 sync and no m2 sync)
> They could have been done in Perl? sure, but it would take me even
> more time to do it and considering that Maven devs are 99% Java
> developers I thought people would be more willing to contrib, fix
> bugs,... in Java
Writing scripts in Java *shudder*. You guys are sick.
I think I'm a Java weenie and then this kind of thing comes along and
I realize I'm some kind of multiple-personalitied
Java-weenie/script-kiddie/god-knows-what-else.
Bet they end up being in Ruby in a years time.
Hen
Re: New sync process in place
Posted by Brett Porter <br...@apache.org>.
On 21/02/2008, at 5:00 AM, Jason van Zyl wrote:
>>
>> I dont think Contegix knows or should know how the mavenupload or
>> sync
>> requests are processed, and do the IT part about processes, disk
>> usage, bandwidth,... And that's still in the crontab, just splitted
>> in
>> portions to reduce breaking points.
>
> I couldn't disagree more. They are a valuable resource and they are
> around 24/7. And they do know about it, they watch everything on the
> machine and pay attention or take action on anything we tell them to.
I think you're misunderstanding each other. I agree with Carlos that
Contegix shouldn't be worrying about processing the requests we get
through JIRA. I agree with Jason that'd be great to have them to make
sure the syncs happen (ie, processes). If they believe the best way
for them to do that requires they write or modify the scripts, that's
probably fine, we can cross that bridge when we come to it.
- Brett
--
Brett Porter
brett@apache.org
http://blogs.exist.com/bporter/
Re: New sync process in place
Posted by Jason van Zyl <ja...@maven.org>.
>
> I dont think Contegix knows or should know how the mavenupload or sync
> requests are processed, and do the IT part about processes, disk
> usage, bandwidth,... And that's still in the crontab, just splitted in
> portions to reduce breaking points.
I couldn't disagree more. They are a valuable resource and they are
around 24/7. And they do know about it, they watch everything on the
machine and pay attention or take action on anything we tell them to.
>
>>
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Apache Maven
jason at sonatype dot com
----------------------------------------------------------
What matters is not ideas, but the people who have them. Good people
can fix bad ideas, but good ideas can't save bad people.
-- Paul Graham
Re: New sync process in place
Posted by Carlos Sanchez <ca...@apache.org>.
On Feb 19, 2008 11:23 PM, Jason van Zyl <ja...@maven.org> wrote:
>
> On 19-Feb-08, at 8:30 PM, Carlos Sanchez wrote:
>
> > the problem with the m2 synchronization scripts is that we need to add
> > one per groupId synced, right now there are 80+ groups to be synced
> > and is a pain to mantain.
>
> And so you had to write it in Java why? I just find it totally odd
> that you ignored the first set of tools written in perl which were
> perfectly fine, into a set of not so good shell scripts which blobbed
> every back together, and now into Java. Perl does fine with threads,
> timeouts, what have you and sysadmin folks have an easier time with it.
The Perl tools were done for maven 1 sync and didn't work for m2 (sync
the whole repo instead of specific groups amongs other problems)
The m2 scripts were the fastest way I had to get m2 syncs in place
when people started asking for them (there were several weeks when we
had m1 sync and no m2 sync)
They could have been done in Perl? sure, but it would take me even
more time to do it and considering that Maven devs are 99% Java
developers I thought people would be more willing to contrib, fix
bugs,... in Java
>
> > It was not easy to see what is synced and
> > from where, while now is a table. I don't imagine the Contegix guys
> > would mess with that one, and that's the only thing converted to java.
> >
>
> Right, perl is no good at text processing. Contegix would maintain
> anything we asked them to. That's part of their job. I also don't
> think it's great having one huge file. I suppose it's a matter of
> preference, but I liked having one set of data per sync.
well, for each person that request a sync I had to download the file,
copy it in the right place and commit
now I just have to do copy & paste and commit, which may not sound
like a lot of work saved, but when there are a ton of issues it will
be faster. Also about 25% of the scripts had typos that I had to
notice manually
>
> > The other script that run all the sync process at once (I imagine is
> > the one you refer to) has the problem as showed up several times
> > before that a slow sync to ibiblio causes all the other syncs to be
> > stopped, and repo1 nor cica.es doesn't get the new data from the
> > partner repos. I have to log in the machine, kill processes,... Now
> > the ibiblio sync happens less often and doesnt block the sync of new
> > releases from partners.
>
> I still don't see how writing a few classes in Java helps there.
that has nothing to do with the java changes, this was just splitting
the monolithic script in its parts:
# sync m1 repos and convert to m2
# sync rewrite rules to m1 repo at ibiblio
# sync m2 repos
# sync central to cica.es mirror
# sync central to ibiblio
>
> You've taken it from a few people + Contegix knowing the bits to you
> alone. Contegix being the important ones. If you had asked them they
> would have been more then happy to make a decent configuration, and
> something with better process control, time outs and simple reporting.
> They whip things like that up all the time and they like doing it and
> they are the ones around 24/7.
I dont think Contegix knows or should know how the mavenupload or sync
requests are processed, and do the IT part about processes, disk
usage, bandwidth,... And that's still in the crontab, just splitted in
portions to reduce breaking points.
>
>
> >
> >
> > On Feb 19, 2008 7:52 PM, Jason van Zyl <ja...@maven.org> wrote:
> >>
> >> On 19-Feb-08, at 7:22 PM, Carlos Sanchez wrote:
> >>
> >>> When I did the previous m2-sync with scripts back in Feb 06 it was
> >>> discussed that was very temporary.
> >>> This version was started June last year and was sitting there
> >>> until I
> >>> found some time lately to finish it.
> >>>
> >>> As nobody really plays with the sync and upload requests I though it
> >>> wouldn't be a problem to try to save myself some time.
> >>> But please raise any concern.
> >>>
> >>
> >> One is that it's now in Java and the Contegix folks do understand the
> >> scripts and in the event they did need to make a change they are now
> >> out of the loop as the maintainers of the machine. What was wrong
> >> with
> >> the scripts?
> >>
> >>
> >>> Thanks
> >>>
> >>>
> >>> On Feb 19, 2008 6:57 PM, Jason van Zyl <ja...@maven.org> wrote:
> >>>> Did I miss some messages actually discussing changing the process?
> >>>>
> >>>>
> >>>> On 19-Feb-08, at 6:45 PM, Carlos Sanchez wrote:
> >>>>
> >>>>> I have finished a new process for Maven 2 repo synchronization,
> >>>>> it's
> >>>>> coded in java and uses a csv file for the list of repositories
> >>>>> https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper/src/bin/synchronize/m2-sync/sync.csv
> >>>>> This will make easier to add new repositories.
> >>>>>
> >>>>> The java code just executes rsync for each repo as the old script
> >>>>> did.
> >>>>>
> >>>>> New features:
> >>>>> * in error both this list and the repo owner will be emailed
> >>>>> Jason, are you receiving maven@maven.org emails? we'd need to
> >>>>> subscribe that address to this list, or change the address to
> >>>>> something else
> >>>>> * there's a 5 minutes timeout for each repo, if it doesnt get
> >>>>> synced
> >>>>> it's reported as an error and it moves on to the next repo
> >>>>>
> >>>>> The code is at https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper
> >>>>>
> >>>>> I've also splitted the big synchronization script in different
> >>>>> pieces
> >>>>> so we sync the m1 repos less often (they are not updated as much),
> >>>>> as
> >>>>> the sync to the ibiblio mirror (takes a long time), and so in case
> >>>>> of
> >>>>> errors in one part (eg ibiblio sync) the others continue (eg. m2
> >>>>> sync
> >>>>> to repo1)
> >>>>>
> >>>>> The new crontab is at
> >>>>> https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper/src/bin/crontab.txt
> >>>>>
> >>>>> Please keep an eye if you make new releases to make sure
> >>>>> everything
> >>>>> works fine and shows up in the central repo and the main mirrors
> >>>>> http://repo1.maven.org/maven2/
> >>>>> http://mirrors.ibiblio.org/pub/mirrors/maven2/
> >>>>> http://ftp.cica.es/mirrors/maven2/
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>> --
> >>>>> I could give you my word as a Spaniard.
> >>>>> No good. I've known too many Spaniards.
> >>>>> -- The Princess Bride
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Jason
> >>>>
> >>>> ----------------------------------------------------------
> >>>> Jason van Zyl
> >>>> Founder, Apache Maven
> >>>> jason at sonatype dot com
> >>>> ----------------------------------------------------------
> >>>>
> >>>> A party which is not afraid of letting culture,
> >>>> business, and welfare go to ruin completely can
> >>>> be omnipotent for a while.
> >>>>
> >>>> -- Jakob Burckhardt
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> I could give you my word as a Spaniard.
> >>> No good. I've known too many Spaniards.
> >>> -- The Princess Bride
> >>
> >> Thanks,
> >>
> >> Jason
> >>
> >> ----------------------------------------------------------
> >> Jason van Zyl
> >> Founder, Apache Maven
> >> jason at sonatype dot com
> >> ----------------------------------------------------------
> >>
> >> Simplex sigillum veri. (Simplicity is the seal of truth.)
> >>
> >>
> >>
> >>
> >
> >
> >
> > --
> > I could give you my word as a Spaniard.
> > No good. I've known too many Spaniards.
> > -- The Princess Bride
>
> Thanks,
>
> Jason
>
> ----------------------------------------------------------
> Jason van Zyl
> Founder, Apache Maven
> jason at sonatype dot com
> ----------------------------------------------------------
>
> A man enjoys his work when he understands the whole and when he
> is responsible for the quality of the whole
>
> -- Christopher Alexander, A Pattern Language
>
>
>
>
--
I could give you my word as a Spaniard.
No good. I've known too many Spaniards.
-- The Princess Bride
Re: New sync process in place
Posted by Jason van Zyl <ja...@maven.org>.
On 19-Feb-08, at 8:30 PM, Carlos Sanchez wrote:
> the problem with the m2 synchronization scripts is that we need to add
> one per groupId synced, right now there are 80+ groups to be synced
> and is a pain to mantain.
And so you had to write it in Java why? I just find it totally odd
that you ignored the first set of tools written in perl which were
perfectly fine, into a set of not so good shell scripts which blobbed
every back together, and now into Java. Perl does fine with threads,
timeouts, what have you and sysadmin folks have an easier time with it.
> It was not easy to see what is synced and
> from where, while now is a table. I don't imagine the Contegix guys
> would mess with that one, and that's the only thing converted to java.
>
Right, perl is no good at text processing. Contegix would maintain
anything we asked them to. That's part of their job. I also don't
think it's great having one huge file. I suppose it's a matter of
preference, but I liked having one set of data per sync.
> The other script that run all the sync process at once (I imagine is
> the one you refer to) has the problem as showed up several times
> before that a slow sync to ibiblio causes all the other syncs to be
> stopped, and repo1 nor cica.es doesn't get the new data from the
> partner repos. I have to log in the machine, kill processes,... Now
> the ibiblio sync happens less often and doesnt block the sync of new
> releases from partners.
I still don't see how writing a few classes in Java helps there.
You've taken it from a few people + Contegix knowing the bits to you
alone. Contegix being the important ones. If you had asked them they
would have been more then happy to make a decent configuration, and
something with better process control, time outs and simple reporting.
They whip things like that up all the time and they like doing it and
they are the ones around 24/7.
>
>
> On Feb 19, 2008 7:52 PM, Jason van Zyl <ja...@maven.org> wrote:
>>
>> On 19-Feb-08, at 7:22 PM, Carlos Sanchez wrote:
>>
>>> When I did the previous m2-sync with scripts back in Feb 06 it was
>>> discussed that was very temporary.
>>> This version was started June last year and was sitting there
>>> until I
>>> found some time lately to finish it.
>>>
>>> As nobody really plays with the sync and upload requests I though it
>>> wouldn't be a problem to try to save myself some time.
>>> But please raise any concern.
>>>
>>
>> One is that it's now in Java and the Contegix folks do understand the
>> scripts and in the event they did need to make a change they are now
>> out of the loop as the maintainers of the machine. What was wrong
>> with
>> the scripts?
>>
>>
>>> Thanks
>>>
>>>
>>> On Feb 19, 2008 6:57 PM, Jason van Zyl <ja...@maven.org> wrote:
>>>> Did I miss some messages actually discussing changing the process?
>>>>
>>>>
>>>> On 19-Feb-08, at 6:45 PM, Carlos Sanchez wrote:
>>>>
>>>>> I have finished a new process for Maven 2 repo synchronization,
>>>>> it's
>>>>> coded in java and uses a csv file for the list of repositories
>>>>> https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper/src/bin/synchronize/m2-sync/sync.csv
>>>>> This will make easier to add new repositories.
>>>>>
>>>>> The java code just executes rsync for each repo as the old script
>>>>> did.
>>>>>
>>>>> New features:
>>>>> * in error both this list and the repo owner will be emailed
>>>>> Jason, are you receiving maven@maven.org emails? we'd need to
>>>>> subscribe that address to this list, or change the address to
>>>>> something else
>>>>> * there's a 5 minutes timeout for each repo, if it doesnt get
>>>>> synced
>>>>> it's reported as an error and it moves on to the next repo
>>>>>
>>>>> The code is at https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper
>>>>>
>>>>> I've also splitted the big synchronization script in different
>>>>> pieces
>>>>> so we sync the m1 repos less often (they are not updated as much),
>>>>> as
>>>>> the sync to the ibiblio mirror (takes a long time), and so in case
>>>>> of
>>>>> errors in one part (eg ibiblio sync) the others continue (eg. m2
>>>>> sync
>>>>> to repo1)
>>>>>
>>>>> The new crontab is at
>>>>> https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper/src/bin/crontab.txt
>>>>>
>>>>> Please keep an eye if you make new releases to make sure
>>>>> everything
>>>>> works fine and shows up in the central repo and the main mirrors
>>>>> http://repo1.maven.org/maven2/
>>>>> http://mirrors.ibiblio.org/pub/mirrors/maven2/
>>>>> http://ftp.cica.es/mirrors/maven2/
>>>>>
>>>>> Thanks
>>>>>
>>>>> --
>>>>> I could give you my word as a Spaniard.
>>>>> No good. I've known too many Spaniards.
>>>>> -- The Princess Bride
>>>>
>>>> Thanks,
>>>>
>>>> Jason
>>>>
>>>> ----------------------------------------------------------
>>>> Jason van Zyl
>>>> Founder, Apache Maven
>>>> jason at sonatype dot com
>>>> ----------------------------------------------------------
>>>>
>>>> A party which is not afraid of letting culture,
>>>> business, and welfare go to ruin completely can
>>>> be omnipotent for a while.
>>>>
>>>> -- Jakob Burckhardt
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> I could give you my word as a Spaniard.
>>> No good. I've known too many Spaniards.
>>> -- The Princess Bride
>>
>> Thanks,
>>
>> Jason
>>
>> ----------------------------------------------------------
>> Jason van Zyl
>> Founder, Apache Maven
>> jason at sonatype dot com
>> ----------------------------------------------------------
>>
>> Simplex sigillum veri. (Simplicity is the seal of truth.)
>>
>>
>>
>>
>
>
>
> --
> I could give you my word as a Spaniard.
> No good. I've known too many Spaniards.
> -- The Princess Bride
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Apache Maven
jason at sonatype dot com
----------------------------------------------------------
A man enjoys his work when he understands the whole and when he
is responsible for the quality of the whole
-- Christopher Alexander, A Pattern Language
Re: New sync process in place
Posted by Carlos Sanchez <ca...@apache.org>.
the problem with the m2 synchronization scripts is that we need to add
one per groupId synced, right now there are 80+ groups to be synced
and is a pain to mantain. It was not easy to see what is synced and
from where, while now is a table. I don't imagine the Contegix guys
would mess with that one, and that's the only thing converted to java.
The other script that run all the sync process at once (I imagine is
the one you refer to) has the problem as showed up several times
before that a slow sync to ibiblio causes all the other syncs to be
stopped, and repo1 nor cica.es doesn't get the new data from the
partner repos. I have to log in the machine, kill processes,... Now
the ibiblio sync happens less often and doesnt block the sync of new
releases from partners.
On Feb 19, 2008 7:52 PM, Jason van Zyl <ja...@maven.org> wrote:
>
> On 19-Feb-08, at 7:22 PM, Carlos Sanchez wrote:
>
> > When I did the previous m2-sync with scripts back in Feb 06 it was
> > discussed that was very temporary.
> > This version was started June last year and was sitting there until I
> > found some time lately to finish it.
> >
> > As nobody really plays with the sync and upload requests I though it
> > wouldn't be a problem to try to save myself some time.
> > But please raise any concern.
> >
>
> One is that it's now in Java and the Contegix folks do understand the
> scripts and in the event they did need to make a change they are now
> out of the loop as the maintainers of the machine. What was wrong with
> the scripts?
>
>
> > Thanks
> >
> >
> > On Feb 19, 2008 6:57 PM, Jason van Zyl <ja...@maven.org> wrote:
> >> Did I miss some messages actually discussing changing the process?
> >>
> >>
> >> On 19-Feb-08, at 6:45 PM, Carlos Sanchez wrote:
> >>
> >>> I have finished a new process for Maven 2 repo synchronization, it's
> >>> coded in java and uses a csv file for the list of repositories
> >>> https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper/src/bin/synchronize/m2-sync/sync.csv
> >>> This will make easier to add new repositories.
> >>>
> >>> The java code just executes rsync for each repo as the old script
> >>> did.
> >>>
> >>> New features:
> >>> * in error both this list and the repo owner will be emailed
> >>> Jason, are you receiving maven@maven.org emails? we'd need to
> >>> subscribe that address to this list, or change the address to
> >>> something else
> >>> * there's a 5 minutes timeout for each repo, if it doesnt get synced
> >>> it's reported as an error and it moves on to the next repo
> >>>
> >>> The code is at https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper
> >>>
> >>> I've also splitted the big synchronization script in different
> >>> pieces
> >>> so we sync the m1 repos less often (they are not updated as much),
> >>> as
> >>> the sync to the ibiblio mirror (takes a long time), and so in case
> >>> of
> >>> errors in one part (eg ibiblio sync) the others continue (eg. m2
> >>> sync
> >>> to repo1)
> >>>
> >>> The new crontab is at
> >>> https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper/src/bin/crontab.txt
> >>>
> >>> Please keep an eye if you make new releases to make sure everything
> >>> works fine and shows up in the central repo and the main mirrors
> >>> http://repo1.maven.org/maven2/
> >>> http://mirrors.ibiblio.org/pub/mirrors/maven2/
> >>> http://ftp.cica.es/mirrors/maven2/
> >>>
> >>> Thanks
> >>>
> >>> --
> >>> I could give you my word as a Spaniard.
> >>> No good. I've known too many Spaniards.
> >>> -- The Princess Bride
> >>
> >> Thanks,
> >>
> >> Jason
> >>
> >> ----------------------------------------------------------
> >> Jason van Zyl
> >> Founder, Apache Maven
> >> jason at sonatype dot com
> >> ----------------------------------------------------------
> >>
> >> A party which is not afraid of letting culture,
> >> business, and welfare go to ruin completely can
> >> be omnipotent for a while.
> >>
> >> -- Jakob Burckhardt
> >>
> >>
> >>
> >>
> >
> >
> >
> > --
> > I could give you my word as a Spaniard.
> > No good. I've known too many Spaniards.
> > -- The Princess Bride
>
> Thanks,
>
> Jason
>
> ----------------------------------------------------------
> Jason van Zyl
> Founder, Apache Maven
> jason at sonatype dot com
> ----------------------------------------------------------
>
> Simplex sigillum veri. (Simplicity is the seal of truth.)
>
>
>
>
--
I could give you my word as a Spaniard.
No good. I've known too many Spaniards.
-- The Princess Bride
Re: New sync process in place
Posted by Jason van Zyl <ja...@maven.org>.
On 19-Feb-08, at 7:22 PM, Carlos Sanchez wrote:
> When I did the previous m2-sync with scripts back in Feb 06 it was
> discussed that was very temporary.
> This version was started June last year and was sitting there until I
> found some time lately to finish it.
>
> As nobody really plays with the sync and upload requests I though it
> wouldn't be a problem to try to save myself some time.
> But please raise any concern.
>
One is that it's now in Java and the Contegix folks do understand the
scripts and in the event they did need to make a change they are now
out of the loop as the maintainers of the machine. What was wrong with
the scripts?
> Thanks
>
>
> On Feb 19, 2008 6:57 PM, Jason van Zyl <ja...@maven.org> wrote:
>> Did I miss some messages actually discussing changing the process?
>>
>>
>> On 19-Feb-08, at 6:45 PM, Carlos Sanchez wrote:
>>
>>> I have finished a new process for Maven 2 repo synchronization, it's
>>> coded in java and uses a csv file for the list of repositories
>>> https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper/src/bin/synchronize/m2-sync/sync.csv
>>> This will make easier to add new repositories.
>>>
>>> The java code just executes rsync for each repo as the old script
>>> did.
>>>
>>> New features:
>>> * in error both this list and the repo owner will be emailed
>>> Jason, are you receiving maven@maven.org emails? we'd need to
>>> subscribe that address to this list, or change the address to
>>> something else
>>> * there's a 5 minutes timeout for each repo, if it doesnt get synced
>>> it's reported as an error and it moves on to the next repo
>>>
>>> The code is at https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper
>>>
>>> I've also splitted the big synchronization script in different
>>> pieces
>>> so we sync the m1 repos less often (they are not updated as much),
>>> as
>>> the sync to the ibiblio mirror (takes a long time), and so in case
>>> of
>>> errors in one part (eg ibiblio sync) the others continue (eg. m2
>>> sync
>>> to repo1)
>>>
>>> The new crontab is at
>>> https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper/src/bin/crontab.txt
>>>
>>> Please keep an eye if you make new releases to make sure everything
>>> works fine and shows up in the central repo and the main mirrors
>>> http://repo1.maven.org/maven2/
>>> http://mirrors.ibiblio.org/pub/mirrors/maven2/
>>> http://ftp.cica.es/mirrors/maven2/
>>>
>>> Thanks
>>>
>>> --
>>> I could give you my word as a Spaniard.
>>> No good. I've known too many Spaniards.
>>> -- The Princess Bride
>>
>> Thanks,
>>
>> Jason
>>
>> ----------------------------------------------------------
>> Jason van Zyl
>> Founder, Apache Maven
>> jason at sonatype dot com
>> ----------------------------------------------------------
>>
>> A party which is not afraid of letting culture,
>> business, and welfare go to ruin completely can
>> be omnipotent for a while.
>>
>> -- Jakob Burckhardt
>>
>>
>>
>>
>
>
>
> --
> I could give you my word as a Spaniard.
> No good. I've known too many Spaniards.
> -- The Princess Bride
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Apache Maven
jason at sonatype dot com
----------------------------------------------------------
Simplex sigillum veri. (Simplicity is the seal of truth.)
Re: New sync process in place
Posted by Carlos Sanchez <ca...@apache.org>.
When I did the previous m2-sync with scripts back in Feb 06 it was
discussed that was very temporary.
This version was started June last year and was sitting there until I
found some time lately to finish it.
As nobody really plays with the sync and upload requests I though it
wouldn't be a problem to try to save myself some time.
But please raise any concern.
Thanks
On Feb 19, 2008 6:57 PM, Jason van Zyl <ja...@maven.org> wrote:
> Did I miss some messages actually discussing changing the process?
>
>
> On 19-Feb-08, at 6:45 PM, Carlos Sanchez wrote:
>
> > I have finished a new process for Maven 2 repo synchronization, it's
> > coded in java and uses a csv file for the list of repositories
> > https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper/src/bin/synchronize/m2-sync/sync.csv
> > This will make easier to add new repositories.
> >
> > The java code just executes rsync for each repo as the old script did.
> >
> > New features:
> > * in error both this list and the repo owner will be emailed
> > Jason, are you receiving maven@maven.org emails? we'd need to
> > subscribe that address to this list, or change the address to
> > something else
> > * there's a 5 minutes timeout for each repo, if it doesnt get synced
> > it's reported as an error and it moves on to the next repo
> >
> > The code is at https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper
> >
> > I've also splitted the big synchronization script in different pieces
> > so we sync the m1 repos less often (they are not updated as much), as
> > the sync to the ibiblio mirror (takes a long time), and so in case of
> > errors in one part (eg ibiblio sync) the others continue (eg. m2 sync
> > to repo1)
> >
> > The new crontab is at
> > https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper/src/bin/crontab.txt
> >
> > Please keep an eye if you make new releases to make sure everything
> > works fine and shows up in the central repo and the main mirrors
> > http://repo1.maven.org/maven2/
> > http://mirrors.ibiblio.org/pub/mirrors/maven2/
> > http://ftp.cica.es/mirrors/maven2/
> >
> > Thanks
> >
> > --
> > I could give you my word as a Spaniard.
> > No good. I've known too many Spaniards.
> > -- The Princess Bride
>
> Thanks,
>
> Jason
>
> ----------------------------------------------------------
> Jason van Zyl
> Founder, Apache Maven
> jason at sonatype dot com
> ----------------------------------------------------------
>
> A party which is not afraid of letting culture,
> business, and welfare go to ruin completely can
> be omnipotent for a while.
>
> -- Jakob Burckhardt
>
>
>
>
--
I could give you my word as a Spaniard.
No good. I've known too many Spaniards.
-- The Princess Bride
Re: New sync process in place
Posted by Jason van Zyl <ja...@maven.org>.
Did I miss some messages actually discussing changing the process?
On 19-Feb-08, at 6:45 PM, Carlos Sanchez wrote:
> I have finished a new process for Maven 2 repo synchronization, it's
> coded in java and uses a csv file for the list of repositories
> https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper/src/bin/synchronize/m2-sync/sync.csv
> This will make easier to add new repositories.
>
> The java code just executes rsync for each repo as the old script did.
>
> New features:
> * in error both this list and the repo owner will be emailed
> Jason, are you receiving maven@maven.org emails? we'd need to
> subscribe that address to this list, or change the address to
> something else
> * there's a 5 minutes timeout for each repo, if it doesnt get synced
> it's reported as an error and it moves on to the next repo
>
> The code is at https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper
>
> I've also splitted the big synchronization script in different pieces
> so we sync the m1 repos less often (they are not updated as much), as
> the sync to the ibiblio mirror (takes a long time), and so in case of
> errors in one part (eg ibiblio sync) the others continue (eg. m2 sync
> to repo1)
>
> The new crontab is at
> https://svn.apache.org/repos/asf/maven/archiva/tools/trunk/maven-meeper/src/bin/crontab.txt
>
> Please keep an eye if you make new releases to make sure everything
> works fine and shows up in the central repo and the main mirrors
> http://repo1.maven.org/maven2/
> http://mirrors.ibiblio.org/pub/mirrors/maven2/
> http://ftp.cica.es/mirrors/maven2/
>
> Thanks
>
> --
> I could give you my word as a Spaniard.
> No good. I've known too many Spaniards.
> -- The Princess Bride
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Apache Maven
jason at sonatype dot com
----------------------------------------------------------
A party which is not afraid of letting culture,
business, and welfare go to ruin completely can
be omnipotent for a while.
-- Jakob Burckhardt