You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openoffice.apache.org by Andre Fischer <af...@a-w-f.de> on 2012/06/04 18:45:25 UTC
The new fetch_tarballs.sh
... is called download_external_libraries.pl.
It is a rewrite in Perl with the following improvements:
- There can be an arbitrary number of download sites. In fact, I have
added all original download sites that I could find (24 of the 38
category-A files, but only 4 of the 14 category-B files).
- There can be several download sites per file. If one does not work
the next is tried. This allows the use of
ooo-extras.apache-extras.org.codespot.com as fallback for the original
site and the SVN repository as a second fallback.
- Each external library can be included or excluded conditionally. For
example the category-B libraries are now loaded only when category-B is
enabled. I used this for one other library, seamonkey. Now it is
loaded only when it is later built (in the moz module).
- On Windows it is much faster when all tar-balls are already present eg
when you just re-run configure with a different set of flags. On my
machine it runs in under 1 second instead of the 5 seconds of
fetch_tarballs.sh: Perl can do most things in process for which
fetch_tarballs.sh had to run external programs. These additional
processes are apparently quite expensive.
Some details:
The download script is main/solenv/bin/download_external_libraries.pl.
You will find it (I hope) well documented.
The data file has a new format and I took the opportunity to rename it
to main/external_libs.lst. It is also documented and should not be too
hard to figure out.
Now the important part:
It did not yet make it the default. The changes are substantial and at
a very central place. In order to activate it just set the environment
variable USE_NEW_EXTLIB_DOWNLOAD to something, before bootstrap is called:
export USE_NEW_EXTLIB_DOWNLOAD=t
I tested it on Windows, Linux, and Mac but there are so many
combinations of configure switches that I may have missed one that does
not yet work with the new download script.
Please try it and tell me about any errors.
I hope this helps in our quest to solve the category-B questions.
Andre
Re: The new fetch_tarballs.sh
Posted by Andre Fischer <af...@a-w-f.de>.
Hi Herbert,
On 05.06.2012 07:25, Herbert Duerr wrote:
> Hi Andre,
>
> thanks for the great work which will help us very much to clean up the
> bootstrap process.
>
>> ... is called download_external_libraries.pl.
>
> I don't like the file names "download_external_libraries.pl" and
> "external_libs.lst" because what is actually downloaded is more than
> just libraries. It already downloads such things a fonts, AFM files and
> some license texts.
>
> We have more than enough cases in our code base where code evolution
> caused a mismatch of names to reality. New functionality that was
> created from scratch should not start with having a misleading name.
>
>> It is a rewrite in Perl with the following improvements:
>> [lots of wonderful benefits snipped]
>> The download script is main/solenv/bin/download_external_libraries.pl.
>
> How about naming that download_external_deps.pl or *_dependencies.pl?
>
>> The data file has a new format and I took the opportunity to rename it
>> to main/external_libs.lst. [...]
>
> How about naming that download_external_deps.lst or *_dependencies.lst?
Good idea. I will rename
download_external_libraries.pl to download_external_dependencies.pl
and
external_libs.lst to external_deps.lst
when I make the new download mechanism the default in a few days.
-Andre
>
>> I hope this helps in our quest to solve the category-B questions.
>
> It sure does. Thanks again!
>
> Herbert
Re: The new fetch_tarballs.sh
Posted by Herbert Duerr <hd...@apache.org>.
Hi Andre,
thanks for the great work which will help us very much to clean up the
bootstrap process.
> ... is called download_external_libraries.pl.
I don't like the file names "download_external_libraries.pl" and
"external_libs.lst" because what is actually downloaded is more than
just libraries. It already downloads such things a fonts, AFM files and
some license texts.
We have more than enough cases in our code base where code evolution
caused a mismatch of names to reality. New functionality that was
created from scratch should not start with having a misleading name.
> It is a rewrite in Perl with the following improvements:
> [lots of wonderful benefits snipped]
> The download script is main/solenv/bin/download_external_libraries.pl.
How about naming that download_external_deps.pl or *_dependencies.pl?
> The data file has a new format and I took the opportunity to rename it
> to main/external_libs.lst. [...]
How about naming that download_external_deps.lst or *_dependencies.lst?
> I hope this helps in our quest to solve the category-B questions.
It sure does. Thanks again!
Herbert
Re: The new fetch_tarballs.sh
Posted by Andre Fischer <af...@a-w-f.de>.
Hi,
I made the new downloading script the default, so that any errors can be
found soon. Maybe it can then be integrated into the 3.4.1 branch.
On 05.06.2012 18:14, Pedro Giffuni wrote:
[...]
>>>
>>> One wish for the future .. hmm.. if I use the external
>>> package (for example for python) it would be nice to
>>> disable downloading the corresponding tarballs too.
I added most of the guards.
>>
>> That is possibly now by just changing the data file. I
>> provided the script. We all can provide the data.
>>
>> There are other things to do, not necessary but nice to
>> have:
>>
>> - Fill in the missing original download sites.
>>
>> - Solve the issues with mismatching MD5 sums.
>>
>> - (mentioned above) Guard more libraries with the same flags
>> that control in their makefiles whether they are build.
>>
>
> Oh, BTW, I didn't carry the checksummed license texts
> in ooo-extras: they are simply useless for building.
> I moved the relevant parts of the texts to the download
> descriptions, it seemed more reasonable to have there.
Good idea.
>
> I am thinking of adding support for binary blobs
> in both rhino and saxon as the Apache policy is OK
> about carrying Java binaries.
Good idea, provided that the license of the libraries allows us to put
the pre-build packages, with the applied patches, on a server.
[...]
-Andre
Re: The new fetch_tarballs.sh
Posted by Pedro Giffuni <pf...@apache.org>.
--- Mar 5/6/12, Andre Fischer <af...@a-w-f.de> ha scritto:
..
> Hi Pedro,
>
> On 04.06.2012 22:05, Pedro Giffuni wrote:
> > Thank you Andre!
> >
> > I normally dislike perl but external_libs.lst surely
> > looks nice now.
>
> Thanks. Well, I dislike Python and did not want to introduce
> yet another scripting language.
>
That's fine.
> >
> > One wish for the future .. hmm.. if I use the external
> > package (for example for python) it would be nice to
> > disable downloading the corresponding tarballs too.
>
> That is possibly now by just changing the data file. I
> provided the script. We all can provide the data.
>
> There are other things to do, not necessary but nice to
> have:
>
> - Fill in the missing original download sites.
>
> - Solve the issues with mismatching MD5 sums.
>
> - (mentioned above) Guard more libraries with the same flags
> that control in their makefiles whether they are build.
>
Oh, BTW, I didn't carry the checksummed license texts
in ooo-extras: they are simply useless for building.
I moved the relevant parts of the texts to the download
descriptions, it seemed more reasonable to have there.
I am thinking of adding support for binary blobs
in both rhino and saxon as the Apache policy is OK
about carrying Java binaries.
Pedro.
> >
> > BTW, should I also upload the Category A tarballs to
> > ooo-extras?
>
> For me that is a question of convenience. Moving them
> (not just copying them) to another site would mean that a
> simple SVN checkout would not put these tarballs on your
> local disk.
>
> Regards,
> Andre
>
> [...]
>
Re: The new fetch_tarballs.sh
Posted by Andre Fischer <af...@a-w-f.de>.
Hi Pedro,
On 04.06.2012 22:05, Pedro Giffuni wrote:
> Thank you Andre!
>
> I normally dislike perl but external_libs.lst surely
> looks nice now.
Thanks. Well, I dislike Python and did not want to introduce yet another
scripting language.
>
> One wish for the future .. hmm.. if I use the external
> package (for example for python) it would be nice to
> disable downloading the corresponding tarballs too.
That is possibly now by just changing the data file. I provided the
script. We all can provide the data.
There are other things to do, not necessary but nice to have:
- Fill in the missing original download sites.
- Solve the issues with mismatching MD5 sums.
- (mentioned above) Guard more libraries with the same flags that
control in their makefiles whether they are build.
>
> BTW, should I also upload the Category A tarballs to
> ooo-extras?
For me that is a question of convenience. Moving them (not just copying
them) to another site would mean that a simple SVN checkout would not
put these tarballs on your local disk.
Regards,
Andre
[...]
Re: The new fetch_tarballs.sh
Posted by Pedro Giffuni <pf...@apache.org>.
Thank you Andre!
I normally dislike perl but external_libs.lst surely
looks nice now.
One wish for the future .. hmm.. if I use the external
package (for example for python) it would be nice to
disable downloading the corresponding tarballs too.
BTW, should I also upload the Category A tarballs to
ooo-extras?
Pedro.
--- Lun 4/6/12, Andre Fischer <af...@a-w-f.de> ha scritto:
> Da: Andre Fischer <af...@a-w-f.de>
> Oggetto: The new fetch_tarballs.sh
> A: ooo-dev@incubator.apache.org
> Data: Lunedì 4 giugno 2012, 11:45
> ... is called
> download_external_libraries.pl.
>
> It is a rewrite in Perl with the following improvements:
>
> - There can be an arbitrary number of download sites.
> In fact, I have added all original download sites that I
> could find (24 of the 38 category-A files, but only 4 of the
> 14 category-B files).
>
> - There can be several download sites per file. If one
> does not work the next is tried. This allows the use
> of ooo-extras.apache-extras.org.codespot.com as fallback for
> the original site and the SVN repository as a second
> fallback.
>
> - Each external library can be included or excluded
> conditionally. For example the category-B libraries
> are now loaded only when category-B is enabled. I used
> this for one other library, seamonkey. Now it is
> loaded only when it is later built (in the moz module).
>
> - On Windows it is much faster when all tar-balls are
> already present eg when you just re-run configure with a
> different set of flags. On my machine it runs in under
> 1 second instead of the 5 seconds of fetch_tarballs.sh: Perl
> can do most things in process for which fetch_tarballs.sh
> had to run external programs. These additional
> processes are apparently quite expensive.
>
>
> Some details:
>
> The download script is
> main/solenv/bin/download_external_libraries.pl. You will
> find it (I hope) well documented.
>
> The data file has a new format and I took the opportunity to
> rename it to main/external_libs.lst. It is also
> documented and should not be too hard to figure out.
>
>
>
> Now the important part:
> It did not yet make it the default. The changes are
> substantial and at a very central place. In order to
> activate it just set the environment variable
> USE_NEW_EXTLIB_DOWNLOAD to something, before bootstrap is
> called:
> export USE_NEW_EXTLIB_DOWNLOAD=t
>
> I tested it on Windows, Linux, and Mac but there are so many
> combinations of configure switches that I may have missed
> one that does not yet work with the new download script.
>
>
>
> Please try it and tell me about any errors.
>
> I hope this helps in our quest to solve the category-B
> questions.
> Andre
>