You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openoffice.apache.org by Damjan Jovanovic <da...@apache.org> on 2016/07/24 17:24:58 UTC
./bootstrap: LWP::Protocol::https replaced by java.net.URLConnection
Hi
Both we and Infra have been battling with the buildbots for far too long,
and the inability to do https:// downloads consistently due to missing
LWP::Protocol::https has been a major thorn in our side.
Perl in AOO is not going well. CPAN modules don't install easily: in both
CentOS 5 and (Infra tells me) in Cygwin some tests fail, preventing
installation. If in 5 months INFRA-11296 has not been closed, and we're
still having mystery problems on the buildbots where Perl modules are
installed but can't be found, and this is even holding up further
development, such as being able to test whether removing VCVARS32.BAT from
the Windows build script fixes the apr build failure and file lock, then I
believe it's past time to tell Perl goodbye!
So as of r1753943, Perl's LWP::UserAgent no longer downloads files. I was
thinking of using a CLI tool instead, something like wget, but that's
another dependency we'd have to document, that may not be available on all
platforms (eg. Mac, OS/2), and that is broken on CentOS 5 for https://
which is what we need it for the most.
So how do we download files now?
Java. Java supports https:// out of the box, is very portable between
operating systems and CPUs, uses its own root CA certificates, is already
used on all the buildbots, and is documented as being a mandatory build
dependency (even though it doesn't seem that way in ./configure). A simple
class main/solenv/javadownloader/AOOJavaDownloader.java gets compiled into
main/solenv/<platform>/class/AOOJavaDownloader.class by the ./bootstrap
script, and then called from Perl's main/solenv/bin/
download_external_dependencies.pl and
main/solenv/bin/modules/ExtensionsLst.pm using system(), in place of
LWP::UserAgent. Internally, it uses java.net.URLConnection for http/https
support, deals with HTTP redirection, also verifying MD5 and SHA1 hashes
like the Perl DownloadFile() function it replaces.
The way it's been set up is a bit of a hack, but it's working phenomenally
well on both FreeBSD and Windows, and with it the Windows aoo-win7 buildbot
has successfully bootstrapped for the first time in memorable history, and
with VCVARS32.BAT eliminated it looks like it may even finish building
successfully!
Damjan
Re: ./bootstrap: LWP::Protocol::https replaced by java.net.URLConnection
Posted by Carl Marcum <cm...@apache.org>.
On 07/24/2016 01:24 PM, Damjan Jovanovic wrote:
> Hi
>
> Both we and Infra have been battling with the buildbots for far too long,
> and the inability to do https:// downloads consistently due to missing
> LWP::Protocol::https has been a major thorn in our side.
>
> Perl in AOO is not going well. CPAN modules don't install easily: in both
> CentOS 5 and (Infra tells me) in Cygwin some tests fail, preventing
> installation. If in 5 months INFRA-11296 has not been closed, and we're
> still having mystery problems on the buildbots where Perl modules are
> installed but can't be found, and this is even holding up further
> development, such as being able to test whether removing VCVARS32.BAT from
> the Windows build script fixes the apr build failure and file lock, then I
> believe it's past time to tell Perl goodbye!
>
> So as of r1753943, Perl's LWP::UserAgent no longer downloads files. I was
> thinking of using a CLI tool instead, something like wget, but that's
> another dependency we'd have to document, that may not be available on all
> platforms (eg. Mac, OS/2), and that is broken on CentOS 5 for https://
> which is what we need it for the most.
>
> So how do we download files now?
>
> Java. Java supports https:// out of the box, is very portable between
> operating systems and CPUs, uses its own root CA certificates, is already
> used on all the buildbots, and is documented as being a mandatory build
> dependency (even though it doesn't seem that way in ./configure). A simple
> class main/solenv/javadownloader/AOOJavaDownloader.java gets compiled into
> main/solenv/<platform>/class/AOOJavaDownloader.class by the ./bootstrap
> script, and then called from Perl's main/solenv/bin/
> download_external_dependencies.pl and
> main/solenv/bin/modules/ExtensionsLst.pm using system(), in place of
> LWP::UserAgent. Internally, it uses java.net.URLConnection for http/https
> support, deals with HTTP redirection, also verifying MD5 and SHA1 hashes
> like the Perl DownloadFile() function it replaces.
>
> The way it's been set up is a bit of a hack, but it's working phenomenally
> well on both FreeBSD and Windows, and with it the Windows aoo-win7 buildbot
> has successfully bootstrapped for the first time in memorable history, and
> with VCVARS32.BAT eliminated it looks like it may even finish building
> successfully!
>
> Damjan
>
That seems like a great solution.
Good luck.
Carl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: ./bootstrap: LWP::Protocol::https replaced by java.net.URLConnection
Posted by Damjan Jovanovic <da...@apache.org>.
On Wed, Jul 27, 2016 at 9:04 PM, Kay Schenk <ks...@apache.org> wrote:
> On Mon, Jul 25, 2016 at 6:38 AM, Damjan Jovanovic <da...@apache.org>
> wrote:
>
> > On Sun, Jul 24, 2016 at 7:43 PM, Patricia Shanahan <pa...@acm.org> wrote:
> >
> > > On 7/24/2016 10:24 AM, Damjan Jovanovic wrote:
> > > ...
> > >
> > >> So how do we download files now?
> > >>
> > >> Java. Java supports https:// out of the box, is very portable between
> > >> operating systems and CPUs, uses its own root CA certificates, is
> > already
> > >> used on all the buildbots, and is documented as being a mandatory
> build
> > >> dependency (even though it doesn't seem that way in ./configure). A
> > simple
> > >> class main/solenv/javadownloader/AOOJavaDownloader.java gets compiled
> > into
> > >> main/solenv/<platform>/class/AOOJavaDownloader.class by the
> ./bootstrap
> > >> script, and then called from Perl's main/solenv/bin/
> > >> download_external_dependencies.pl and
> > >> main/solenv/bin/modules/ExtensionsLst.pm using system(), in place of
> > >> LWP::UserAgent. Internally, it uses java.net.URLConnection for
> > http/https
> > >> support, deals with HTTP redirection, also verifying MD5 and SHA1
> hashes
> > >> like the Perl DownloadFile() function it replaces.
> > >>
> > >> The way it's been set up is a bit of a hack, but it's working
> > phenomenally
> > >> well on both FreeBSD and Windows, and with it the Windows aoo-win7
> > >> buildbot
> > >> has successfully bootstrapped for the first time in memorable history,
> > and
> > >> with VCVARS32.BAT eliminated it looks like it may even finish building
> > >> successfully!
> > >>
> > >
> > > This sounds very promising. I hope it works.
> > >
> >
> > Every nightly buildbot bootstraps now and every *nix buildbot among them
> > builds successfully. I consider the patch a complete success.
> >
> > The snapshot buildbots still can't bootstrap unless they have
> > LWP::Protocol::https installed - not until we make another snapshot for
> > them with r1753943 merged.
> >
> > It's just Windows we have to fix now. See the other thread.
> >
>
> Hi. OK, and maybe a dumb question. The Perl procedure also checks the MD5
> checksums. Would a new java download procedure also be setup to do this?
>
>
Absolutely. It already does.
Re: ./bootstrap: LWP::Protocol::https replaced by java.net.URLConnection
Posted by Kay Schenk <ks...@apache.org>.
On Mon, Jul 25, 2016 at 6:38 AM, Damjan Jovanovic <da...@apache.org> wrote:
> On Sun, Jul 24, 2016 at 7:43 PM, Patricia Shanahan <pa...@acm.org> wrote:
>
> > On 7/24/2016 10:24 AM, Damjan Jovanovic wrote:
> > ...
> >
> >> So how do we download files now?
> >>
> >> Java. Java supports https:// out of the box, is very portable between
> >> operating systems and CPUs, uses its own root CA certificates, is
> already
> >> used on all the buildbots, and is documented as being a mandatory build
> >> dependency (even though it doesn't seem that way in ./configure). A
> simple
> >> class main/solenv/javadownloader/AOOJavaDownloader.java gets compiled
> into
> >> main/solenv/<platform>/class/AOOJavaDownloader.class by the ./bootstrap
> >> script, and then called from Perl's main/solenv/bin/
> >> download_external_dependencies.pl and
> >> main/solenv/bin/modules/ExtensionsLst.pm using system(), in place of
> >> LWP::UserAgent. Internally, it uses java.net.URLConnection for
> http/https
> >> support, deals with HTTP redirection, also verifying MD5 and SHA1 hashes
> >> like the Perl DownloadFile() function it replaces.
> >>
> >> The way it's been set up is a bit of a hack, but it's working
> phenomenally
> >> well on both FreeBSD and Windows, and with it the Windows aoo-win7
> >> buildbot
> >> has successfully bootstrapped for the first time in memorable history,
> and
> >> with VCVARS32.BAT eliminated it looks like it may even finish building
> >> successfully!
> >>
> >
> > This sounds very promising. I hope it works.
> >
>
> Every nightly buildbot bootstraps now and every *nix buildbot among them
> builds successfully. I consider the patch a complete success.
>
> The snapshot buildbots still can't bootstrap unless they have
> LWP::Protocol::https installed - not until we make another snapshot for
> them with r1753943 merged.
>
> It's just Windows we have to fix now. See the other thread.
>
Hi. OK, and maybe a dumb question. The Perl procedure also checks the MD5
checksums. Would a new java download procedure also be setup to do this?
--
--------------------------------------------------------------------------------------------------------
Kay Schenk@Apache OpenOffice
“Things work out best for those who make the best of how things work out.”
-- John Wooden
Re: ./bootstrap: LWP::Protocol::https replaced by java.net.URLConnection
Posted by Damjan Jovanovic <da...@apache.org>.
On Sun, Jul 24, 2016 at 7:43 PM, Patricia Shanahan <pa...@acm.org> wrote:
> On 7/24/2016 10:24 AM, Damjan Jovanovic wrote:
> ...
>
>> So how do we download files now?
>>
>> Java. Java supports https:// out of the box, is very portable between
>> operating systems and CPUs, uses its own root CA certificates, is already
>> used on all the buildbots, and is documented as being a mandatory build
>> dependency (even though it doesn't seem that way in ./configure). A simple
>> class main/solenv/javadownloader/AOOJavaDownloader.java gets compiled into
>> main/solenv/<platform>/class/AOOJavaDownloader.class by the ./bootstrap
>> script, and then called from Perl's main/solenv/bin/
>> download_external_dependencies.pl and
>> main/solenv/bin/modules/ExtensionsLst.pm using system(), in place of
>> LWP::UserAgent. Internally, it uses java.net.URLConnection for http/https
>> support, deals with HTTP redirection, also verifying MD5 and SHA1 hashes
>> like the Perl DownloadFile() function it replaces.
>>
>> The way it's been set up is a bit of a hack, but it's working phenomenally
>> well on both FreeBSD and Windows, and with it the Windows aoo-win7
>> buildbot
>> has successfully bootstrapped for the first time in memorable history, and
>> with VCVARS32.BAT eliminated it looks like it may even finish building
>> successfully!
>>
>
> This sounds very promising. I hope it works.
>
Every nightly buildbot bootstraps now and every *nix buildbot among them
builds successfully. I consider the patch a complete success.
The snapshot buildbots still can't bootstrap unless they have
LWP::Protocol::https installed - not until we make another snapshot for
them with r1753943 merged.
It's just Windows we have to fix now. See the other thread.
Re: ./bootstrap: LWP::Protocol::https replaced by
java.net.URLConnection
Posted by Patricia Shanahan <pa...@acm.org>.
On 7/24/2016 10:24 AM, Damjan Jovanovic wrote:
...
> So how do we download files now?
>
> Java. Java supports https:// out of the box, is very portable between
> operating systems and CPUs, uses its own root CA certificates, is already
> used on all the buildbots, and is documented as being a mandatory build
> dependency (even though it doesn't seem that way in ./configure). A simple
> class main/solenv/javadownloader/AOOJavaDownloader.java gets compiled into
> main/solenv/<platform>/class/AOOJavaDownloader.class by the ./bootstrap
> script, and then called from Perl's main/solenv/bin/
> download_external_dependencies.pl and
> main/solenv/bin/modules/ExtensionsLst.pm using system(), in place of
> LWP::UserAgent. Internally, it uses java.net.URLConnection for http/https
> support, deals with HTTP redirection, also verifying MD5 and SHA1 hashes
> like the Perl DownloadFile() function it replaces.
>
> The way it's been set up is a bit of a hack, but it's working phenomenally
> well on both FreeBSD and Windows, and with it the Windows aoo-win7 buildbot
> has successfully bootstrapped for the first time in memorable history, and
> with VCVARS32.BAT eliminated it looks like it may even finish building
> successfully!
This sounds very promising. I hope it works.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org