Installer and MD5 Checksums

[Alex Harui <ah...@adobe.com>]

The Installer 3.1 uses MD5 checksums on many of its downloads to verify
their accuracy.  The checksums for Google Closure Library, Adobe AIR SDK
and playerglobal.swc change every once in a while and that breaks installs
until we find out, download the file, compute the new checksum and update
the sdk-installer-config-4.0.xml file on the web-site.

Is there a way we can automate that?  The Adobe AIR SDK is over 200MB
(times 2 platforms) so it would be a lot of bandwidth to keep downloading
it.  Is it possible to implement quicker check the way browsers verify
what is in their caches?

Thoughts?
-Alex

Re: Installer and MD5 Checksums

[Alex Harui <ah...@adobe.com>]

On 7/13/14 9:52 AM, "OmPrakash Muppirala" <bi...@gmail.com> wrote:

>On Sat, Jul 12, 2014 at 9:11 PM, Alex Harui <ah...@adobe.com> wrote:
>
>>
>>
>> On 7/12/14 11:05 AM, "OmPrakash Muppirala" <bi...@gmail.com> wrote:
>>
>> >On Sat, Jul 12, 2014 at 10:55 AM, Alex Harui <ah...@adobe.com> wrote:
>> >
>> >> The Installer 3.1 uses MD5 checksums on many of its downloads to
>>verify
>> >> their accuracy.  The checksums for Google Closure Library, Adobe AIR
>>SDK
>> >> and playerglobal.swc change every once in a while and that breaks
>> >>installs
>> >> until we find out, download the file, compute the new checksum and
>> >>update
>> >> the sdk-installer-config-4.0.xml file on the web-site.
>> >>
>> >> Is there a way we can automate that?  The Adobe AIR SDK is over 200MB
>> >> (times 2 platforms) so it would be a lot of bandwidth to keep
>> >>downloading
>> >> it.  Is it possible to implement quicker check the way browsers
>>verify
>> >> what is in their caches?
>> >>
>> >>
>> >Technically, yes we can automate it by creating a Jenkins job on one of
>> >our
>> >servers to just calculate a checksum of a downloaded file.  We can make
>> >the
>> >.MD5 file publicly available that the Installer could use.  This would
>>be
>> >a
>> >one time set up.  We can probably run the job once a day.
>> That could add up to 1GB per day or more?  Would that go beyond some
>>limit
>> allowed by Azure provider?
>>
>>
>1GB per day in terms of bandwidth or storage?  Either way we have to try
>it
>out and see what happens.
Bandwidth.  2 platforms times how many AIR releases times 260MB per
release.

>
>
>>  >
>> >But I question the need for this. Normally checksums are created from
>>the
>> >source.  The act of downloading to your computer (manually or
>> >automatically
>> >via a job) could corrupt the file   If you compute the checksum on a
>> >corrupted file and use that to verify subsequent downloads from a
>>server,
>> >that defeats the purpose.
>> >
>> >Either we ask Adobe, Google etc. to upload checksums which we can
>>directly
>> >download from the Installer and verify them.  Or we skip verification
>>of
>> >checksums of binaries that don't originate from ASF servers/mirrors.
>> Bad downloads have been a significant problem.  And with the install
>> scripts caching downloads, I wanted to verify the download before
>>sticking
>> into the cache.
>
>
>
>How about unzipping and and doing a brain-dead check for certain files?
> Like air-config.xml or something like that.  And only then stick it into
>the cache?
Ant usually throws an exception trying to unzip.

-ALex

Re: Installer and MD5 Checksums

[OmPrakash Muppirala <bi...@gmail.com>]

On Sat, Jul 12, 2014 at 9:11 PM, Alex Harui <ah...@adobe.com> wrote:

>
>
> On 7/12/14 11:05 AM, "OmPrakash Muppirala" <bi...@gmail.com> wrote:
>
> >On Sat, Jul 12, 2014 at 10:55 AM, Alex Harui <ah...@adobe.com> wrote:
> >
> >> The Installer 3.1 uses MD5 checksums on many of its downloads to verify
> >> their accuracy.  The checksums for Google Closure Library, Adobe AIR SDK
> >> and playerglobal.swc change every once in a while and that breaks
> >>installs
> >> until we find out, download the file, compute the new checksum and
> >>update
> >> the sdk-installer-config-4.0.xml file on the web-site.
> >>
> >> Is there a way we can automate that?  The Adobe AIR SDK is over 200MB
> >> (times 2 platforms) so it would be a lot of bandwidth to keep
> >>downloading
> >> it.  Is it possible to implement quicker check the way browsers verify
> >> what is in their caches?
> >>
> >>
> >Technically, yes we can automate it by creating a Jenkins job on one of
> >our
> >servers to just calculate a checksum of a downloaded file.  We can make
> >the
> >.MD5 file publicly available that the Installer could use.  This would be
> >a
> >one time set up.  We can probably run the job once a day.
> That could add up to 1GB per day or more?  Would that go beyond some limit
> allowed by Azure provider?
>
>
1GB per day in terms of bandwidth or storage?  Either way we have to try it
out and see what happens.


>  >
> >But I question the need for this. Normally checksums are created from the
> >source.  The act of downloading to your computer (manually or
> >automatically
> >via a job) could corrupt the file   If you compute the checksum on a
> >corrupted file and use that to verify subsequent downloads from a server,
> >that defeats the purpose.
> >
> >Either we ask Adobe, Google etc. to upload checksums which we can directly
> >download from the Installer and verify them.  Or we skip verification of
> >checksums of binaries that don't originate from ASF servers/mirrors.
> Bad downloads have been a significant problem.  And with the install
> scripts caching downloads, I wanted to verify the download before sticking
> into the cache.



How about unzipping and and doing a brain-dead check for certain files?
 Like air-config.xml or something like that.  And only then stick it into
the cache?


> If there's a better way to do that in Ant, I'm open to
> it, although I guess that will mean even more time before we ever ship
> another release.
>

Nope.  I dont want to add anything to the list for this release.  Lets get
this one out the door first :-)

Thanks,
Om


>
> -Alex
>
>

Re: Installer and MD5 Checksums

[Alex Harui <ah...@adobe.com>]

On 7/12/14 11:05 AM, "OmPrakash Muppirala" <bi...@gmail.com> wrote:

>On Sat, Jul 12, 2014 at 10:55 AM, Alex Harui <ah...@adobe.com> wrote:
>
>> The Installer 3.1 uses MD5 checksums on many of its downloads to verify
>> their accuracy.  The checksums for Google Closure Library, Adobe AIR SDK
>> and playerglobal.swc change every once in a while and that breaks
>>installs
>> until we find out, download the file, compute the new checksum and
>>update
>> the sdk-installer-config-4.0.xml file on the web-site.
>>
>> Is there a way we can automate that?  The Adobe AIR SDK is over 200MB
>> (times 2 platforms) so it would be a lot of bandwidth to keep
>>downloading
>> it.  Is it possible to implement quicker check the way browsers verify
>> what is in their caches?
>>
>>
>Technically, yes we can automate it by creating a Jenkins job on one of
>our
>servers to just calculate a checksum of a downloaded file.  We can make
>the
>.MD5 file publicly available that the Installer could use.  This would be
>a
>one time set up.  We can probably run the job once a day.
That could add up to 1GB per day or more?  Would that go beyond some limit
allowed by Azure provider?

>
>But I question the need for this. Normally checksums are created from the
>source.  The act of downloading to your computer (manually or
>automatically
>via a job) could corrupt the file   If you compute the checksum on a
>corrupted file and use that to verify subsequent downloads from a server,
>that defeats the purpose.
>
>Either we ask Adobe, Google etc. to upload checksums which we can directly
>download from the Installer and verify them.  Or we skip verification of
>checksums of binaries that don't originate from ASF servers/mirrors.
Bad downloads have been a significant problem.  And with the install
scripts caching downloads, I wanted to verify the download before sticking
into the cache.  If there's a better way to do that in Ant, I'm open to
it, although I guess that will mean even more time before we ever ship
another release.

-Alex

Re: Installer and MD5 Checksums

[OmPrakash Muppirala <bi...@gmail.com>]

On Sat, Jul 12, 2014 at 10:55 AM, Alex Harui <ah...@adobe.com> wrote:

> The Installer 3.1 uses MD5 checksums on many of its downloads to verify
> their accuracy.  The checksums for Google Closure Library, Adobe AIR SDK
> and playerglobal.swc change every once in a while and that breaks installs
> until we find out, download the file, compute the new checksum and update
> the sdk-installer-config-4.0.xml file on the web-site.
>
> Is there a way we can automate that?  The Adobe AIR SDK is over 200MB
> (times 2 platforms) so it would be a lot of bandwidth to keep downloading
> it.  Is it possible to implement quicker check the way browsers verify
> what is in their caches?
>
>
Technically, yes we can automate it by creating a Jenkins job on one of our
servers to just calculate a checksum of a downloaded file.  We can make the
.MD5 file publicly available that the Installer could use.  This would be a
one time set up.  We can probably run the job once a day.

But I question the need for this. Normally checksums are created from the
source.  The act of downloading to your computer (manually or automatically
via a job) could corrupt the file   If you compute the checksum on a
corrupted file and use that to verify subsequent downloads from a server,
that defeats the purpose.

Either we ask Adobe, Google etc. to upload checksums which we can directly
download from the Installer and verify them.  Or we skip verification of
checksums of binaries that don't originate from ASF servers/mirrors.

Thanks,
Om



> Thoughts?
> -Alex
>
>