You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openoffice.apache.org by Dave Fisher <da...@comcast.net> on 2012/08/26 21:20:44 UTC
Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
Hi,
We need to do more work to have proper compliance with Apache Infrastructure policy in managing external dependencies.
I may not be precisely correct and am looking for confirmation, but In general i think we need to
(1) Completely avoid using svn.apache.org. I don't think we are allowed to do this even as a backup URL.
(2) Use mirrors or maven for ASF dependencies where we use the current release. If we use mirrors then archive.apache.org should be the backup for the mirror so that we aren't in trouble if the project has a release. If a maven repository were used then there would be no issue.
(3) If we use mirrors then we should allow the user to choose which mirror.
If we decide to take the time to go the maven route. I can use the example of ant and maven repos from the Apache POI build.xml.
Notes about maven repos. Infra [1], maven central [2] and example of an externally hosted repo [3]
This area needs careful attention.
The current script is here: main/solenv/bin/download_external_dependencies.pl
Regards,
Dave
[1] http://apache.org/dev/repository-faq.html and
[2] http://maven.apache.org/guides/mini/guide-central-repository-upload.html
[3] http://repo.maven.apache.org/maven2/javax/activation/activation/1.0.2/activation-1.0.2.pom
On Aug 26, 2012, at 11:58 AM, wave@apache.org wrote:
> Author: wave
> Date: Sun Aug 26 18:58:08 2012
> New Revision: 1377482
>
> URL: http://svn.apache.org/viewvc?rev=1377482&view=rev
> Log:
> one more small step to infra compliance. still to do removing use of svn as a backup and for current releases of ASF software the archive is not proper - either a mirror or the maven repository is required.
>
> Modified:
> incubator/ooo/trunk/main/external_deps.lst
>
> Modified: incubator/ooo/trunk/main/external_deps.lst
> URL: http://svn.apache.org/viewvc/incubator/ooo/trunk/main/external_deps.lst?rev=1377482&r1=1377481&r2=1377482&view=diff
> ==============================================================================
> --- incubator/ooo/trunk/main/external_deps.lst (original)
> +++ incubator/ooo/trunk/main/external_deps.lst Sun Aug 26 18:58:08 2012
> @@ -72,7 +72,7 @@ if ( true )
> if (SOLAR_JAVA == TRUE)
> MD5 = 17960f35b2239654ba608cf1f3e256b3
> name = lucene-2.9.4-src.tar.gz
> - URL1 = http://www.us.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
> + URL1 = http://archive.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
> URL2 = $(OOO_EXTRAS)$(MD5)-$(name)
> # Fall back to a version in SVN from a previous revsion.
> URL3 = http://svn.apache.org/repos/asf/!svn/bc/1337615/incubator/ooo/trunk/ext_sources/$(MD5)-$(name)
>
>
RE: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
Posted by "Dennis E. Hamilton" <or...@apache.org>.
I concur with Rob's analysis here (quoted below).
It occurs to me that there is another spanner in the works. This will impact whatever the signing process is.
Namely: Not only does the installer need to be signed, but those *installed* components (e.g., .exe and .dll files) that can bear embedded, OS-recognized signatures need to be signed if products of the build from source. In the case of bundled third-party components that are installed as-is, they may need to be signed by their originator as well.
This is part of the authenticity provision. It has an important provision in determining whether or not a component has been altered or replaced after installation. (In the case of drivers and other components, the OS may be fussier than that.) This is also important if component-level replacement by patches is introduced.
I don't know that all of this has to be swallowed at once. I is going to be a factor in the future and it would be good to take preparatory action.
I expect that the bar will be raised on extensions too, and that might be handled by upgrading .oxt to use ODF 1.2 packaging, relying on the digital signature provisions that are already there for packages and special components such as scripts and macros.
- Dennis
-----Original Message-----
From: Rob Weir [mailto:robweir@apache.org]
Sent: Sunday, August 26, 2012 13:47
To: ooo-dev@incubator.apache.org
Subject: Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
[ ... ]
My point is that if we seek to have build-bot automated signed
binaries it is entirely foreseeable that what we're proposing to do
with 3rd party dependencies will also be out of policy, due to the
security concerns. To do this we'll need to do more than just link to
random sites on the web for dependencies.
[ ... ]
Look at Andre's note from August 3rd. First locations searched for
dependencies is the original website where it is hosted. Then it
checks Apache Extras. Then it checks SVN. Taking SVN out of the
loop solves one issue. But we still have the other issue -- I think
Dennis groks this as well -- that our primary search location is less
secure than the alternatives.
[ ... ], two relatively simple improvements:
1) Search Apache Extras first
or
2) Have the current script verify a detached signature rather than
relying on MD5 hashes
[ ... ]
Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
Posted by Pedro Giffuni <pf...@apache.org>.
----- Original Message -----
,,,
>>
>> Please take the time to study the Maven project before you attempt to
> reinvent it. [1]
>>
>
> I think one could fairly look at the relative creation dates and see
> that Maven reinvented OpenOffice's dependency tracking system ;-)
>
Both are reinvention of FreeBSD's ports tree (where checksums have
been working very reliably for both security and consistency). The
question is which is more reliable and flexible.
If we use maven will that involve an extra dependency on Java?
Cheers,
Pedro.
Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
Posted by Dave Fisher <da...@comcast.net>.
On Aug 26, 2012, at 2:49 PM, Rob Weir wrote:
> On Sun, Aug 26, 2012 at 5:20 PM, Pedro Giffuni <pf...@apache.org> wrote:
>> BTW,
>>
>> ----- Original Message -----
>> ...
>>>>
>>>> This is already part of the current process. The signatures are in
>>> download_external_dependencies.pl. The Central Maven Repository uses these as
>>> well.
>>>>
>>>
>>> Those are MD5 hashes, not signatures. MD5 has been broken since 1996:
>>>
>>> http://en.wikipedia.org/wiki/MD5#Collision_vulnerabilities
>>>
>>
>> We can simply replace MD5 with SHA256 (Apache-Extras
>> generates SHA1).
>>
>
> Good enough for Bitcoins...
r1377529 removes SVN!
I'm checking to see what is Maven central - http://search.maven.org/
Regards,
Dave
>
>> Pedro.
Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
Posted by Rob Weir <ro...@apache.org>.
On Sun, Aug 26, 2012 at 5:20 PM, Pedro Giffuni <pf...@apache.org> wrote:
> BTW,
>
> ----- Original Message -----
> ...
>>>
>>> This is already part of the current process. The signatures are in
>> download_external_dependencies.pl. The Central Maven Repository uses these as
>> well.
>>>
>>
>> Those are MD5 hashes, not signatures. MD5 has been broken since 1996:
>>
>> http://en.wikipedia.org/wiki/MD5#Collision_vulnerabilities
>>
>
> We can simply replace MD5 with SHA256 (Apache-Extras
> generates SHA1).
>
Good enough for Bitcoins...
> Pedro.
Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
Posted by Pedro Giffuni <pf...@apache.org>.
BTW,
----- Original Message -----
...
>>
>> This is already part of the current process. The signatures are in
> download_external_dependencies.pl. The Central Maven Repository uses these as
> well.
>>
>
> Those are MD5 hashes, not signatures. MD5 has been broken since 1996:
>
> http://en.wikipedia.org/wiki/MD5#Collision_vulnerabilities
>
We can simply replace MD5 with SHA256 (Apache-Extras
generates SHA1).
Pedro.
Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
Posted by Dave Fisher <da...@comcast.net>.
On Aug 26, 2012, at 1:46 PM, Rob Weir wrote:
> On Sun, Aug 26, 2012 at 3:50 PM, Dave Fisher <da...@comcast.net> wrote:
>>
>> On Aug 26, 2012, at 12:38 PM, Rob Weir wrote:
>>
>>> On Sun, Aug 26, 2012 at 3:20 PM, Dave Fisher <da...@comcast.net> wrote:
>>>> Hi,
>>>>
>>>> We need to do more work to have proper compliance with Apache Infrastructure policy in managing external dependencies.
>>>>
>>>> I may not be precisely correct and am looking for confirmation, but In general i think we need to
>>>>
>>>> (1) Completely avoid using svn.apache.org. I don't think we are allowed to do this even as a backup URL.
>>>>
>>>> (2) Use mirrors or maven for ASF dependencies where we use the current release. If we use mirrors then archive.apache.org should be the backup for the mirror so that we aren't in trouble if the project has a release. If a maven repository were used then there would be no issue.
>>>>
>>>> (3) If we use mirrors then we should allow the user to choose which mirror.
>>>>
>>>> If we decide to take the time to go the maven route. I can use the example of ant and maven repos from the Apache POI build.xml.
>>>>
>>>> Notes about maven repos. Infra [1], maven central [2] and example of an externally hosted repo [3]
>>>>
>>>> This area needs careful attention.
>>>>
>>
>> Please take the time to study the Maven project before you attempt to reinvent it. [1]
>>
>
> I think one could fairly look at the relative creation dates and see
> that Maven reinvented OpenOffice's dependency tracking system ;-)
>
>>> Note that this move is exactly the wrong thing to do if we want have
>>> buildbots build binaries that are assumed to be safe and therefore
>>> signable. Instead of the security and verifiability of ASF-run host,
>>> we're putting the dependencies off to a dozen different remote sites,
>>> with no visibility into their site's mechanisms for vetting changes,
>>> access controls, auditability of changes, even basics like ensuring
>>> domain names are renewed and not poached by others.
>>>
>>> Do we really think other websites are as secure as the ones that Infra
>>> operates? If so we should move the source code to the other sites as
>>> well, right?
>>
>> This is not the question. My comments were about using the ASF sites properly. Joe has told us that we are doing this out of policy.
>>
>
> My point is that if we seek to have build-bot automated signed
> binaries it is entirely foreseeable that what we're proposing to do
> with 3rd party dependencies will also be out of policy, due to the
> security concerns. To do this we'll need to do more than just link to
> random sites on the web for dependencies.
Yes and this is why I brought up Maven.
Here is an example of how our process is fragile if the dependency is another Apache project. Every so often there are new release or retirement. When this happens the release is moved to the archives.
This problem will happen for one of our dependencies on 12/31/12 when the whole Tomcat 5.5.x branch is retired.
With a Maven repository everything goes by an id. I'll research the rules to see if it is a solution to all of our dependency issues.
>
>>>
>>> No easy resolution of this, but we might mitigate the risk by putting
>>> all of the dependencies to Apache-Extras and load from there
>>> primarily. And if at all possible make sure all change notifications
>>> from there get echoed to the ooo-committs lis. We have a better
>>> chance of exercising now screwing up if we control rather than having
>>> multiple 3rd parties control.
>>
>> We need to differentiate between ASF projects and third party projects. Look at the current process before you go nuts assuming that everything is changing.
>>
>
> Look at Andre's note from August 3rd. First locations searched for
> dependencies is the original website where it is hosted. Then it
> checks Apache Extras. Then it checks SVN. Taking SVN out of the
> loop solves one issue.
> But we still have the other issue -- I think
> Dennis groks this as well -- that our primary search location is less
> secure than the alternatives.
I get it too. apache extras is one easy place for that, but a Maven repository may be another. Where and how that gets seeded is our business.
>
>>>
>>> Another option would be to use cryptographic means to ensure the
>>> integrity of the remote dependencies, e.g., detached signatures. That
>>> doesn't protect us from another website going down, temporarily or
>>> permanently, but it does allow us to verify that what we are
>>> downloading has not been tampered with.
>>
>> This is already part of the current process. The signatures are in download_external_dependencies.pl. The Central Maven Repository uses these as well.
>>
>
> Those are MD5 hashes, not signatures. MD5 has been broken since 1996:
>
> http://en.wikipedia.org/wiki/MD5#Collision_vulnerabilities
>
> Again, two relatively simple improvements:
>
> 1) Search Apache Extras first
>
> or
>
> 2) Have the current script verify a detached signature rather than
> relying on MD5 hashes
>
>
> Note: the MD5 hashes were OK previously, since we also maintained
> control over the actually files. This was true under Sun/Oracle, as
> well as earlier in the Podling. But the risk profile changes
> significantly once we start retrieving these files from a website over
> which we have no control.
Or no total control. Let's see.
We should probably go through each external dependency and fully classify it to make sure we understand every case properly. The information is in the dependency list.
I'll do a review.
Regards,
Dave
>
> -Rob
>
>
>
>> [1] http://maven.apache.org/
>>
>>>
>>
>>
>>>
>>> -Rob
>>>
>>>
>>>> The current script is here: main/solenv/bin/download_external_dependencies.pl
>>>>
>>>> Regards,
>>>> Dave
>>>>
>>>> [1] http://apache.org/dev/repository-faq.html and
>>>> [2] http://maven.apache.org/guides/mini/guide-central-repository-upload.html
>>>> [3] http://repo.maven.apache.org/maven2/javax/activation/activation/1.0.2/activation-1.0.2.pom
>>>>
>>>>
>>>> On Aug 26, 2012, at 11:58 AM, wave@apache.org wrote:
>>>>
>>>>> Author: wave
>>>>> Date: Sun Aug 26 18:58:08 2012
>>>>> New Revision: 1377482
>>>>>
>>>>> URL: http://svn.apache.org/viewvc?rev=1377482&view=rev
>>>>> Log:
>>>>> one more small step to infra compliance. still to do removing use of svn as a backup and for current releases of ASF software the archive is not proper - either a mirror or the maven repository is required.
>>>>>
>>>>> Modified:
>>>>> incubator/ooo/trunk/main/external_deps.lst
>>>>>
>>>>> Modified: incubator/ooo/trunk/main/external_deps.lst
>>>>> URL: http://svn.apache.org/viewvc/incubator/ooo/trunk/main/external_deps.lst?rev=1377482&r1=1377481&r2=1377482&view=diff
>>>>> ==============================================================================
>>>>> --- incubator/ooo/trunk/main/external_deps.lst (original)
>>>>> +++ incubator/ooo/trunk/main/external_deps.lst Sun Aug 26 18:58:08 2012
>>>>> @@ -72,7 +72,7 @@ if ( true )
>>>>> if (SOLAR_JAVA == TRUE)
>>>>> MD5 = 17960f35b2239654ba608cf1f3e256b3
>>>>> name = lucene-2.9.4-src.tar.gz
>>>>> - URL1 = http://www.us.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
>>>>> + URL1 = http://archive.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
>>>>> URL2 = $(OOO_EXTRAS)$(MD5)-$(name)
>>>>> # Fall back to a version in SVN from a previous revsion.
>>>>> URL3 = http://svn.apache.org/repos/asf/!svn/bc/1337615/incubator/ooo/trunk/ext_sources/$(MD5)-$(name)
>>>>>
>>>>>
>>>>
>>
Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
Posted by Rob Weir <ro...@apache.org>.
On Sun, Aug 26, 2012 at 3:50 PM, Dave Fisher <da...@comcast.net> wrote:
>
> On Aug 26, 2012, at 12:38 PM, Rob Weir wrote:
>
>> On Sun, Aug 26, 2012 at 3:20 PM, Dave Fisher <da...@comcast.net> wrote:
>>> Hi,
>>>
>>> We need to do more work to have proper compliance with Apache Infrastructure policy in managing external dependencies.
>>>
>>> I may not be precisely correct and am looking for confirmation, but In general i think we need to
>>>
>>> (1) Completely avoid using svn.apache.org. I don't think we are allowed to do this even as a backup URL.
>>>
>>> (2) Use mirrors or maven for ASF dependencies where we use the current release. If we use mirrors then archive.apache.org should be the backup for the mirror so that we aren't in trouble if the project has a release. If a maven repository were used then there would be no issue.
>>>
>>> (3) If we use mirrors then we should allow the user to choose which mirror.
>>>
>>> If we decide to take the time to go the maven route. I can use the example of ant and maven repos from the Apache POI build.xml.
>>>
>>> Notes about maven repos. Infra [1], maven central [2] and example of an externally hosted repo [3]
>>>
>>> This area needs careful attention.
>>>
>
> Please take the time to study the Maven project before you attempt to reinvent it. [1]
>
I think one could fairly look at the relative creation dates and see
that Maven reinvented OpenOffice's dependency tracking system ;-)
>> Note that this move is exactly the wrong thing to do if we want have
>> buildbots build binaries that are assumed to be safe and therefore
>> signable. Instead of the security and verifiability of ASF-run host,
>> we're putting the dependencies off to a dozen different remote sites,
>> with no visibility into their site's mechanisms for vetting changes,
>> access controls, auditability of changes, even basics like ensuring
>> domain names are renewed and not poached by others.
>>
>> Do we really think other websites are as secure as the ones that Infra
>> operates? If so we should move the source code to the other sites as
>> well, right?
>
> This is not the question. My comments were about using the ASF sites properly. Joe has told us that we are doing this out of policy.
>
My point is that if we seek to have build-bot automated signed
binaries it is entirely foreseeable that what we're proposing to do
with 3rd party dependencies will also be out of policy, due to the
security concerns. To do this we'll need to do more than just link to
random sites on the web for dependencies.
>>
>> No easy resolution of this, but we might mitigate the risk by putting
>> all of the dependencies to Apache-Extras and load from there
>> primarily. And if at all possible make sure all change notifications
>> from there get echoed to the ooo-committs lis. We have a better
>> chance of exercising now screwing up if we control rather than having
>> multiple 3rd parties control.
>
> We need to differentiate between ASF projects and third party projects. Look at the current process before you go nuts assuming that everything is changing.
>
Look at Andre's note from August 3rd. First locations searched for
dependencies is the original website where it is hosted. Then it
checks Apache Extras. Then it checks SVN. Taking SVN out of the
loop solves one issue. But we still have the other issue -- I think
Dennis groks this as well -- that our primary search location is less
secure than the alternatives.
>>
>> Another option would be to use cryptographic means to ensure the
>> integrity of the remote dependencies, e.g., detached signatures. That
>> doesn't protect us from another website going down, temporarily or
>> permanently, but it does allow us to verify that what we are
>> downloading has not been tampered with.
>
> This is already part of the current process. The signatures are in download_external_dependencies.pl. The Central Maven Repository uses these as well.
>
Those are MD5 hashes, not signatures. MD5 has been broken since 1996:
http://en.wikipedia.org/wiki/MD5#Collision_vulnerabilities
Again, two relatively simple improvements:
1) Search Apache Extras first
or
2) Have the current script verify a detached signature rather than
relying on MD5 hashes
Note: the MD5 hashes were OK previously, since we also maintained
control over the actually files. This was true under Sun/Oracle, as
well as earlier in the Podling. But the risk profile changes
significantly once we start retrieving these files from a website over
which we have no control.
-Rob
> [1] http://maven.apache.org/
>
>>
>
>
>>
>> -Rob
>>
>>
>>> The current script is here: main/solenv/bin/download_external_dependencies.pl
>>>
>>> Regards,
>>> Dave
>>>
>>> [1] http://apache.org/dev/repository-faq.html and
>>> [2] http://maven.apache.org/guides/mini/guide-central-repository-upload.html
>>> [3] http://repo.maven.apache.org/maven2/javax/activation/activation/1.0.2/activation-1.0.2.pom
>>>
>>>
>>> On Aug 26, 2012, at 11:58 AM, wave@apache.org wrote:
>>>
>>>> Author: wave
>>>> Date: Sun Aug 26 18:58:08 2012
>>>> New Revision: 1377482
>>>>
>>>> URL: http://svn.apache.org/viewvc?rev=1377482&view=rev
>>>> Log:
>>>> one more small step to infra compliance. still to do removing use of svn as a backup and for current releases of ASF software the archive is not proper - either a mirror or the maven repository is required.
>>>>
>>>> Modified:
>>>> incubator/ooo/trunk/main/external_deps.lst
>>>>
>>>> Modified: incubator/ooo/trunk/main/external_deps.lst
>>>> URL: http://svn.apache.org/viewvc/incubator/ooo/trunk/main/external_deps.lst?rev=1377482&r1=1377481&r2=1377482&view=diff
>>>> ==============================================================================
>>>> --- incubator/ooo/trunk/main/external_deps.lst (original)
>>>> +++ incubator/ooo/trunk/main/external_deps.lst Sun Aug 26 18:58:08 2012
>>>> @@ -72,7 +72,7 @@ if ( true )
>>>> if (SOLAR_JAVA == TRUE)
>>>> MD5 = 17960f35b2239654ba608cf1f3e256b3
>>>> name = lucene-2.9.4-src.tar.gz
>>>> - URL1 = http://www.us.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
>>>> + URL1 = http://archive.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
>>>> URL2 = $(OOO_EXTRAS)$(MD5)-$(name)
>>>> # Fall back to a version in SVN from a previous revsion.
>>>> URL3 = http://svn.apache.org/repos/asf/!svn/bc/1337615/incubator/ooo/trunk/ext_sources/$(MD5)-$(name)
>>>>
>>>>
>>>
>
Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
Posted by Dave Fisher <da...@comcast.net>.
On Aug 26, 2012, at 12:38 PM, Rob Weir wrote:
> On Sun, Aug 26, 2012 at 3:20 PM, Dave Fisher <da...@comcast.net> wrote:
>> Hi,
>>
>> We need to do more work to have proper compliance with Apache Infrastructure policy in managing external dependencies.
>>
>> I may not be precisely correct and am looking for confirmation, but In general i think we need to
>>
>> (1) Completely avoid using svn.apache.org. I don't think we are allowed to do this even as a backup URL.
>>
>> (2) Use mirrors or maven for ASF dependencies where we use the current release. If we use mirrors then archive.apache.org should be the backup for the mirror so that we aren't in trouble if the project has a release. If a maven repository were used then there would be no issue.
>>
>> (3) If we use mirrors then we should allow the user to choose which mirror.
>>
>> If we decide to take the time to go the maven route. I can use the example of ant and maven repos from the Apache POI build.xml.
>>
>> Notes about maven repos. Infra [1], maven central [2] and example of an externally hosted repo [3]
>>
>> This area needs careful attention.
>>
Please take the time to study the Maven project before you attempt to reinvent it. [1]
> Note that this move is exactly the wrong thing to do if we want have
> buildbots build binaries that are assumed to be safe and therefore
> signable. Instead of the security and verifiability of ASF-run host,
> we're putting the dependencies off to a dozen different remote sites,
> with no visibility into their site's mechanisms for vetting changes,
> access controls, auditability of changes, even basics like ensuring
> domain names are renewed and not poached by others.
>
> Do we really think other websites are as secure as the ones that Infra
> operates? If so we should move the source code to the other sites as
> well, right?
This is not the question. My comments were about using the ASF sites properly. Joe has told us that we are doing this out of policy.
>
> No easy resolution of this, but we might mitigate the risk by putting
> all of the dependencies to Apache-Extras and load from there
> primarily. And if at all possible make sure all change notifications
> from there get echoed to the ooo-committs lis. We have a better
> chance of exercising now screwing up if we control rather than having
> multiple 3rd parties control.
We need to differentiate between ASF projects and third party projects. Look at the current process before you go nuts assuming that everything is changing.
>
> Another option would be to use cryptographic means to ensure the
> integrity of the remote dependencies, e.g., detached signatures. That
> doesn't protect us from another website going down, temporarily or
> permanently, but it does allow us to verify that what we are
> downloading has not been tampered with.
This is already part of the current process. The signatures are in download_external_dependencies.pl. The Central Maven Repository uses these as well.
[1] http://maven.apache.org/
>
>
> -Rob
>
>
>> The current script is here: main/solenv/bin/download_external_dependencies.pl
>>
>> Regards,
>> Dave
>>
>> [1] http://apache.org/dev/repository-faq.html and
>> [2] http://maven.apache.org/guides/mini/guide-central-repository-upload.html
>> [3] http://repo.maven.apache.org/maven2/javax/activation/activation/1.0.2/activation-1.0.2.pom
>>
>>
>> On Aug 26, 2012, at 11:58 AM, wave@apache.org wrote:
>>
>>> Author: wave
>>> Date: Sun Aug 26 18:58:08 2012
>>> New Revision: 1377482
>>>
>>> URL: http://svn.apache.org/viewvc?rev=1377482&view=rev
>>> Log:
>>> one more small step to infra compliance. still to do removing use of svn as a backup and for current releases of ASF software the archive is not proper - either a mirror or the maven repository is required.
>>>
>>> Modified:
>>> incubator/ooo/trunk/main/external_deps.lst
>>>
>>> Modified: incubator/ooo/trunk/main/external_deps.lst
>>> URL: http://svn.apache.org/viewvc/incubator/ooo/trunk/main/external_deps.lst?rev=1377482&r1=1377481&r2=1377482&view=diff
>>> ==============================================================================
>>> --- incubator/ooo/trunk/main/external_deps.lst (original)
>>> +++ incubator/ooo/trunk/main/external_deps.lst Sun Aug 26 18:58:08 2012
>>> @@ -72,7 +72,7 @@ if ( true )
>>> if (SOLAR_JAVA == TRUE)
>>> MD5 = 17960f35b2239654ba608cf1f3e256b3
>>> name = lucene-2.9.4-src.tar.gz
>>> - URL1 = http://www.us.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
>>> + URL1 = http://archive.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
>>> URL2 = $(OOO_EXTRAS)$(MD5)-$(name)
>>> # Fall back to a version in SVN from a previous revsion.
>>> URL3 = http://svn.apache.org/repos/asf/!svn/bc/1337615/incubator/ooo/trunk/ext_sources/$(MD5)-$(name)
>>>
>>>
>>
Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
Posted by Dave Fisher <da...@comcast.net>.
On Aug 26, 2012, at 1:08 PM, Dennis E. Hamilton wrote:
> I haven't said anything about modifications to an upstream source. That's an entirely different problem.
Yes. I think that is what concerns Rob. Maintenance of necessary forks and protection from lost dependencies.
>
> I'm talking about dependencies on someone else's binaries of any kind.
We need to be concerned about required binary dependencies. We should definitely keep copies of these, and build from our copies.
Regards,
Dave
>
> - Dennis
>
> PS: In my own analysis, I probably should have mentioned bundled extensions too, although that seems to be a rather AOO-specific case. At some point, the entire extension provenance and authenticity case *will* come under scrutiny.
>
> -----Original Message-----
> From: Dave Fisher [mailto:dave2wave@comcast.net]
> Sent: Sunday, August 26, 2012 12:57
> To: ooo-dev@incubator.apache.org
> Subject: Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
>
>
> On Aug 26, 2012, at 12:47 PM, Dennis E. Hamilton wrote:
>
>> +1 on preserving the provenance and integrity of binary dependencies.
>>
>> I'd go with external signatures *and* hosting the specific artifacts on a reliable ASF location for preservation along with all ASF project sources that depended on those specific artifacts in any sort of review, release of authenticated binaries, etc.
>
> There is nothing to say that we can't make our modifications with code from svn and base code stored in a reliable location. We can then push this to either extras and/or maven central.
>
> Regards,
> Dave
>
>>
>> - Dennis
>>
>> -----Original Message-----
>> From: Rob Weir [mailto:robweir@apache.org]
>> Sent: Sunday, August 26, 2012 12:38
>> To: ooo-dev@incubator.apache.org
>> Subject: Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
>>
>> On Sun, Aug 26, 2012 at 3:20 PM, Dave Fisher <da...@comcast.net> wrote:
>>> Hi,
>>>
>>> We need to do more work to have proper compliance with Apache Infrastructure policy in managing external dependencies.
>>>
>>> I may not be precisely correct and am looking for confirmation, but In general i think we need to
>>>
>>> (1) Completely avoid using svn.apache.org. I don't think we are allowed to do this even as a backup URL.
>>>
>>> (2) Use mirrors or maven for ASF dependencies where we use the current release. If we use mirrors then archive.apache.org should be the backup for the mirror so that we aren't in trouble if the project has a release. If a maven repository were used then there would be no issue.
>>>
>>> (3) If we use mirrors then we should allow the user to choose which mirror.
>>>
>>> If we decide to take the time to go the maven route. I can use the example of ant and maven repos from the Apache POI build.xml.
>>>
>>> Notes about maven repos. Infra [1], maven central [2] and example of an externally hosted repo [3]
>>>
>>> This area needs careful attention.
>>>
>>
>> Note that this move is exactly the wrong thing to do if we want have
>> buildbots build binaries that are assumed to be safe and therefore
>> signable. Instead of the security and verifiability of ASF-run host,
>> we're putting the dependencies off to a dozen different remote sites,
>> with no visibility into their site's mechanisms for vetting changes,
>> access controls, auditability of changes, even basics like ensuring
>> domain names are renewed and not poached by others.
>>
>> Do we really think other websites are as secure as the ones that Infra
>> operates? If so we should move the source code to the other sites as
>> well, right?
>>
>> No easy resolution of this, but we might mitigate the risk by putting
>> all of the dependencies to Apache-Extras and load from there
>> primarily. And if at all possible make sure all change notifications
>> from there get echoed to the ooo-committs lis. We have a better
>> chance of exercising now screwing up if we control rather than having
>> multiple 3rd parties control.
>>
>> Another option would be to use cryptographic means to ensure the
>> integrity of the remote dependencies, e.g., detached signatures. That
>> doesn't protect us from another website going down, temporarily or
>> permanently, but it does allow us to verify that what we are
>> downloading has not been tampered with.
>>
>> -Rob
>>
>>
>>> The current script is here: main/solenv/bin/download_external_dependencies.pl
>>>
>>> Regards,
>>> Dave
>>>
>>> [1] http://apache.org/dev/repository-faq.html and
>>> [2] http://maven.apache.org/guides/mini/guide-central-repository-upload.html
>>> [3] http://repo.maven.apache.org/maven2/javax/activation/activation/1.0.2/activation-1.0.2.pom
>>>
>>>
>>> On Aug 26, 2012, at 11:58 AM, wave@apache.org wrote:
>>>
>>>> Author: wave
>>>> Date: Sun Aug 26 18:58:08 2012
>>>> New Revision: 1377482
>>>>
>>>> URL: http://svn.apache.org/viewvc?rev=1377482&view=rev
>>>> Log:
>>>> one more small step to infra compliance. still to do removing use of svn as a backup and for current releases of ASF software the archive is not proper - either a mirror or the maven repository is required.
>>>>
>>>> Modified:
>>>> incubator/ooo/trunk/main/external_deps.lst
>>>>
>>>> Modified: incubator/ooo/trunk/main/external_deps.lst
>>>> URL: http://svn.apache.org/viewvc/incubator/ooo/trunk/main/external_deps.lst?rev=1377482&r1=1377481&r2=1377482&view=diff
>>>> ==============================================================================
>>>> --- incubator/ooo/trunk/main/external_deps.lst (original)
>>>> +++ incubator/ooo/trunk/main/external_deps.lst Sun Aug 26 18:58:08 2012
>>>> @@ -72,7 +72,7 @@ if ( true )
>>>> if (SOLAR_JAVA == TRUE)
>>>> MD5 = 17960f35b2239654ba608cf1f3e256b3
>>>> name = lucene-2.9.4-src.tar.gz
>>>> - URL1 = http://www.us.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
>>>> + URL1 = http://archive.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
>>>> URL2 = $(OOO_EXTRAS)$(MD5)-$(name)
>>>> # Fall back to a version in SVN from a previous revsion.
>>>> URL3 = http://svn.apache.org/repos/asf/!svn/bc/1337615/incubator/ooo/trunk/ext_sources/$(MD5)-$(name)
>>>>
>>>>
>>>
>>
>
RE: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
Posted by "Dennis E. Hamilton" <or...@apache.org>.
I haven't said anything about modifications to an upstream source. That's an entirely different problem.
I'm talking about dependencies on someone else's binaries of any kind.
- Dennis
PS: In my own analysis, I probably should have mentioned bundled extensions too, although that seems to be a rather AOO-specific case. At some point, the entire extension provenance and authenticity case *will* come under scrutiny.
-----Original Message-----
From: Dave Fisher [mailto:dave2wave@comcast.net]
Sent: Sunday, August 26, 2012 12:57
To: ooo-dev@incubator.apache.org
Subject: Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
On Aug 26, 2012, at 12:47 PM, Dennis E. Hamilton wrote:
> +1 on preserving the provenance and integrity of binary dependencies.
>
> I'd go with external signatures *and* hosting the specific artifacts on a reliable ASF location for preservation along with all ASF project sources that depended on those specific artifacts in any sort of review, release of authenticated binaries, etc.
There is nothing to say that we can't make our modifications with code from svn and base code stored in a reliable location. We can then push this to either extras and/or maven central.
Regards,
Dave
>
> - Dennis
>
> -----Original Message-----
> From: Rob Weir [mailto:robweir@apache.org]
> Sent: Sunday, August 26, 2012 12:38
> To: ooo-dev@incubator.apache.org
> Subject: Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
>
> On Sun, Aug 26, 2012 at 3:20 PM, Dave Fisher <da...@comcast.net> wrote:
>> Hi,
>>
>> We need to do more work to have proper compliance with Apache Infrastructure policy in managing external dependencies.
>>
>> I may not be precisely correct and am looking for confirmation, but In general i think we need to
>>
>> (1) Completely avoid using svn.apache.org. I don't think we are allowed to do this even as a backup URL.
>>
>> (2) Use mirrors or maven for ASF dependencies where we use the current release. If we use mirrors then archive.apache.org should be the backup for the mirror so that we aren't in trouble if the project has a release. If a maven repository were used then there would be no issue.
>>
>> (3) If we use mirrors then we should allow the user to choose which mirror.
>>
>> If we decide to take the time to go the maven route. I can use the example of ant and maven repos from the Apache POI build.xml.
>>
>> Notes about maven repos. Infra [1], maven central [2] and example of an externally hosted repo [3]
>>
>> This area needs careful attention.
>>
>
> Note that this move is exactly the wrong thing to do if we want have
> buildbots build binaries that are assumed to be safe and therefore
> signable. Instead of the security and verifiability of ASF-run host,
> we're putting the dependencies off to a dozen different remote sites,
> with no visibility into their site's mechanisms for vetting changes,
> access controls, auditability of changes, even basics like ensuring
> domain names are renewed and not poached by others.
>
> Do we really think other websites are as secure as the ones that Infra
> operates? If so we should move the source code to the other sites as
> well, right?
>
> No easy resolution of this, but we might mitigate the risk by putting
> all of the dependencies to Apache-Extras and load from there
> primarily. And if at all possible make sure all change notifications
> from there get echoed to the ooo-committs lis. We have a better
> chance of exercising now screwing up if we control rather than having
> multiple 3rd parties control.
>
> Another option would be to use cryptographic means to ensure the
> integrity of the remote dependencies, e.g., detached signatures. That
> doesn't protect us from another website going down, temporarily or
> permanently, but it does allow us to verify that what we are
> downloading has not been tampered with.
>
> -Rob
>
>
>> The current script is here: main/solenv/bin/download_external_dependencies.pl
>>
>> Regards,
>> Dave
>>
>> [1] http://apache.org/dev/repository-faq.html and
>> [2] http://maven.apache.org/guides/mini/guide-central-repository-upload.html
>> [3] http://repo.maven.apache.org/maven2/javax/activation/activation/1.0.2/activation-1.0.2.pom
>>
>>
>> On Aug 26, 2012, at 11:58 AM, wave@apache.org wrote:
>>
>>> Author: wave
>>> Date: Sun Aug 26 18:58:08 2012
>>> New Revision: 1377482
>>>
>>> URL: http://svn.apache.org/viewvc?rev=1377482&view=rev
>>> Log:
>>> one more small step to infra compliance. still to do removing use of svn as a backup and for current releases of ASF software the archive is not proper - either a mirror or the maven repository is required.
>>>
>>> Modified:
>>> incubator/ooo/trunk/main/external_deps.lst
>>>
>>> Modified: incubator/ooo/trunk/main/external_deps.lst
>>> URL: http://svn.apache.org/viewvc/incubator/ooo/trunk/main/external_deps.lst?rev=1377482&r1=1377481&r2=1377482&view=diff
>>> ==============================================================================
>>> --- incubator/ooo/trunk/main/external_deps.lst (original)
>>> +++ incubator/ooo/trunk/main/external_deps.lst Sun Aug 26 18:58:08 2012
>>> @@ -72,7 +72,7 @@ if ( true )
>>> if (SOLAR_JAVA == TRUE)
>>> MD5 = 17960f35b2239654ba608cf1f3e256b3
>>> name = lucene-2.9.4-src.tar.gz
>>> - URL1 = http://www.us.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
>>> + URL1 = http://archive.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
>>> URL2 = $(OOO_EXTRAS)$(MD5)-$(name)
>>> # Fall back to a version in SVN from a previous revsion.
>>> URL3 = http://svn.apache.org/repos/asf/!svn/bc/1337615/incubator/ooo/trunk/ext_sources/$(MD5)-$(name)
>>>
>>>
>>
>
Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
Posted by Dave Fisher <da...@comcast.net>.
On Aug 26, 2012, at 12:47 PM, Dennis E. Hamilton wrote:
> +1 on preserving the provenance and integrity of binary dependencies.
>
> I'd go with external signatures *and* hosting the specific artifacts on a reliable ASF location for preservation along with all ASF project sources that depended on those specific artifacts in any sort of review, release of authenticated binaries, etc.
There is nothing to say that we can't make our modifications with code from svn and base code stored in a reliable location. We can then push this to either extras and/or maven central.
Regards,
Dave
>
> - Dennis
>
> -----Original Message-----
> From: Rob Weir [mailto:robweir@apache.org]
> Sent: Sunday, August 26, 2012 12:38
> To: ooo-dev@incubator.apache.org
> Subject: Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
>
> On Sun, Aug 26, 2012 at 3:20 PM, Dave Fisher <da...@comcast.net> wrote:
>> Hi,
>>
>> We need to do more work to have proper compliance with Apache Infrastructure policy in managing external dependencies.
>>
>> I may not be precisely correct and am looking for confirmation, but In general i think we need to
>>
>> (1) Completely avoid using svn.apache.org. I don't think we are allowed to do this even as a backup URL.
>>
>> (2) Use mirrors or maven for ASF dependencies where we use the current release. If we use mirrors then archive.apache.org should be the backup for the mirror so that we aren't in trouble if the project has a release. If a maven repository were used then there would be no issue.
>>
>> (3) If we use mirrors then we should allow the user to choose which mirror.
>>
>> If we decide to take the time to go the maven route. I can use the example of ant and maven repos from the Apache POI build.xml.
>>
>> Notes about maven repos. Infra [1], maven central [2] and example of an externally hosted repo [3]
>>
>> This area needs careful attention.
>>
>
> Note that this move is exactly the wrong thing to do if we want have
> buildbots build binaries that are assumed to be safe and therefore
> signable. Instead of the security and verifiability of ASF-run host,
> we're putting the dependencies off to a dozen different remote sites,
> with no visibility into their site's mechanisms for vetting changes,
> access controls, auditability of changes, even basics like ensuring
> domain names are renewed and not poached by others.
>
> Do we really think other websites are as secure as the ones that Infra
> operates? If so we should move the source code to the other sites as
> well, right?
>
> No easy resolution of this, but we might mitigate the risk by putting
> all of the dependencies to Apache-Extras and load from there
> primarily. And if at all possible make sure all change notifications
> from there get echoed to the ooo-committs lis. We have a better
> chance of exercising now screwing up if we control rather than having
> multiple 3rd parties control.
>
> Another option would be to use cryptographic means to ensure the
> integrity of the remote dependencies, e.g., detached signatures. That
> doesn't protect us from another website going down, temporarily or
> permanently, but it does allow us to verify that what we are
> downloading has not been tampered with.
>
> -Rob
>
>
>> The current script is here: main/solenv/bin/download_external_dependencies.pl
>>
>> Regards,
>> Dave
>>
>> [1] http://apache.org/dev/repository-faq.html and
>> [2] http://maven.apache.org/guides/mini/guide-central-repository-upload.html
>> [3] http://repo.maven.apache.org/maven2/javax/activation/activation/1.0.2/activation-1.0.2.pom
>>
>>
>> On Aug 26, 2012, at 11:58 AM, wave@apache.org wrote:
>>
>>> Author: wave
>>> Date: Sun Aug 26 18:58:08 2012
>>> New Revision: 1377482
>>>
>>> URL: http://svn.apache.org/viewvc?rev=1377482&view=rev
>>> Log:
>>> one more small step to infra compliance. still to do removing use of svn as a backup and for current releases of ASF software the archive is not proper - either a mirror or the maven repository is required.
>>>
>>> Modified:
>>> incubator/ooo/trunk/main/external_deps.lst
>>>
>>> Modified: incubator/ooo/trunk/main/external_deps.lst
>>> URL: http://svn.apache.org/viewvc/incubator/ooo/trunk/main/external_deps.lst?rev=1377482&r1=1377481&r2=1377482&view=diff
>>> ==============================================================================
>>> --- incubator/ooo/trunk/main/external_deps.lst (original)
>>> +++ incubator/ooo/trunk/main/external_deps.lst Sun Aug 26 18:58:08 2012
>>> @@ -72,7 +72,7 @@ if ( true )
>>> if (SOLAR_JAVA == TRUE)
>>> MD5 = 17960f35b2239654ba608cf1f3e256b3
>>> name = lucene-2.9.4-src.tar.gz
>>> - URL1 = http://www.us.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
>>> + URL1 = http://archive.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
>>> URL2 = $(OOO_EXTRAS)$(MD5)-$(name)
>>> # Fall back to a version in SVN from a previous revsion.
>>> URL3 = http://svn.apache.org/repos/asf/!svn/bc/1337615/incubator/ooo/trunk/ext_sources/$(MD5)-$(name)
>>>
>>>
>>
>
RE: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
Posted by "Dennis E. Hamilton" <or...@apache.org>.
+1 on preserving the provenance and integrity of binary dependencies.
I'd go with external signatures *and* hosting the specific artifacts on a reliable ASF location for preservation along with all ASF project sources that depended on those specific artifacts in any sort of review, release of authenticated binaries, etc.
- Dennis
-----Original Message-----
From: Rob Weir [mailto:robweir@apache.org]
Sent: Sunday, August 26, 2012 12:38
To: ooo-dev@incubator.apache.org
Subject: Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
On Sun, Aug 26, 2012 at 3:20 PM, Dave Fisher <da...@comcast.net> wrote:
> Hi,
>
> We need to do more work to have proper compliance with Apache Infrastructure policy in managing external dependencies.
>
> I may not be precisely correct and am looking for confirmation, but In general i think we need to
>
> (1) Completely avoid using svn.apache.org. I don't think we are allowed to do this even as a backup URL.
>
> (2) Use mirrors or maven for ASF dependencies where we use the current release. If we use mirrors then archive.apache.org should be the backup for the mirror so that we aren't in trouble if the project has a release. If a maven repository were used then there would be no issue.
>
> (3) If we use mirrors then we should allow the user to choose which mirror.
>
> If we decide to take the time to go the maven route. I can use the example of ant and maven repos from the Apache POI build.xml.
>
> Notes about maven repos. Infra [1], maven central [2] and example of an externally hosted repo [3]
>
> This area needs careful attention.
>
Note that this move is exactly the wrong thing to do if we want have
buildbots build binaries that are assumed to be safe and therefore
signable. Instead of the security and verifiability of ASF-run host,
we're putting the dependencies off to a dozen different remote sites,
with no visibility into their site's mechanisms for vetting changes,
access controls, auditability of changes, even basics like ensuring
domain names are renewed and not poached by others.
Do we really think other websites are as secure as the ones that Infra
operates? If so we should move the source code to the other sites as
well, right?
No easy resolution of this, but we might mitigate the risk by putting
all of the dependencies to Apache-Extras and load from there
primarily. And if at all possible make sure all change notifications
from there get echoed to the ooo-committs lis. We have a better
chance of exercising now screwing up if we control rather than having
multiple 3rd parties control.
Another option would be to use cryptographic means to ensure the
integrity of the remote dependencies, e.g., detached signatures. That
doesn't protect us from another website going down, temporarily or
permanently, but it does allow us to verify that what we are
downloading has not been tampered with.
-Rob
> The current script is here: main/solenv/bin/download_external_dependencies.pl
>
> Regards,
> Dave
>
> [1] http://apache.org/dev/repository-faq.html and
> [2] http://maven.apache.org/guides/mini/guide-central-repository-upload.html
> [3] http://repo.maven.apache.org/maven2/javax/activation/activation/1.0.2/activation-1.0.2.pom
>
>
> On Aug 26, 2012, at 11:58 AM, wave@apache.org wrote:
>
>> Author: wave
>> Date: Sun Aug 26 18:58:08 2012
>> New Revision: 1377482
>>
>> URL: http://svn.apache.org/viewvc?rev=1377482&view=rev
>> Log:
>> one more small step to infra compliance. still to do removing use of svn as a backup and for current releases of ASF software the archive is not proper - either a mirror or the maven repository is required.
>>
>> Modified:
>> incubator/ooo/trunk/main/external_deps.lst
>>
>> Modified: incubator/ooo/trunk/main/external_deps.lst
>> URL: http://svn.apache.org/viewvc/incubator/ooo/trunk/main/external_deps.lst?rev=1377482&r1=1377481&r2=1377482&view=diff
>> ==============================================================================
>> --- incubator/ooo/trunk/main/external_deps.lst (original)
>> +++ incubator/ooo/trunk/main/external_deps.lst Sun Aug 26 18:58:08 2012
>> @@ -72,7 +72,7 @@ if ( true )
>> if (SOLAR_JAVA == TRUE)
>> MD5 = 17960f35b2239654ba608cf1f3e256b3
>> name = lucene-2.9.4-src.tar.gz
>> - URL1 = http://www.us.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
>> + URL1 = http://archive.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
>> URL2 = $(OOO_EXTRAS)$(MD5)-$(name)
>> # Fall back to a version in SVN from a previous revsion.
>> URL3 = http://svn.apache.org/repos/asf/!svn/bc/1337615/incubator/ooo/trunk/ext_sources/$(MD5)-$(name)
>>
>>
>
Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
Posted by Rob Weir <ro...@apache.org>.
n Sun, Aug 26, 2012 at 3:54 PM, Pedro Giffuni <pf...@apache.org> wrote:
>
>
> ----- Original Message -----
>
>>
>> On Sun, Aug 26, 2012 at 3:20 PM, Dave Fisher <da...@comcast.net>
>> wrote:
>>> Hi,
>>>
>>> We need to do more work to have proper compliance with Apache
>> Infrastructure policy in managing external dependencies.
>>>
>>> I may not be precisely correct and am looking for confirmation, but In
>> general i think we need to
>>>
>>> (1) Completely avoid using svn.apache.org. I don't think we are allowed
>> to do this even as a backup URL.
>>>
>>> (2) Use mirrors or maven for ASF dependencies where we use the current
>> release. If we use mirrors then archive.apache.org should be the backup for the
>> mirror so that we aren't in trouble if the project has a release. If a maven
>> repository were used then there would be no issue.
>>>
>>> (3) If we use mirrors then we should allow the user to choose which mirror.
>>>
>>> If we decide to take the time to go the maven route. I can use the example
>> of ant and maven repos from the Apache POI build.xml.
>>>
>>> Notes about maven repos. Infra [1], maven central [2] and example of an
>> externally hosted repo [3]
>>>
>>> This area needs careful attention.
>>>
>>
>> Note that this move is exactly the wrong thing to do if we want have
>> buildbots build binaries that are assumed to be safe and therefore
>> signable. Instead of the security and verifiability of ASF-run host,
>> we're putting the dependencies off to a dozen different remote sites,
>> with no visibility into their site's mechanisms for vetting changes,
>> access controls, auditability of changes, even basics like ensuring
>> domain names are renewed and not poached by others.
>>
>
> This is not true. We use checksums precisely to certify the source tarballs
> haven't been tampered with.
>
If we think that then we're fooling ourselves. Checksums are
effective for detecting accidental corruption during transmission, or
for detecting when someone casually modifies a file. It does very
little to prevent us against malicious changes. To be specific: One
can make whatever changes one wishes in a file to insert malicious
code, and then via brute force (or better) make other changes to an
unrelated file in the archive in order to achieve the desired hash
collision. Is it computationally expensive? Yes. Is it impossible?
No. Does a binary used by 10 million people present an incentive that
would attract that kind of attention? I don't want to find out.
This is why Apache releases have the detached digital signature as
well. It gives a much higher degree of safety.
>
>
>> Do we really think other websites are as secure as the ones that Infra
>> operates? If so we should move the source code to the other sites as
>> well, right?
>>
>
> Yes, upstream providers generate checksums and do proper
> maintainance of their packages but perhaps more importantly by
> working with them we can leverage their mirrors more efficiently.
>
> I agree with Dave; we have to get rid of those tarballs
> in SVN. Python 2.6.1 will be axed today, BTW.
>
> I haven't looked at Maven but it sounds like something
> we should look at..
>
> Pedro.
Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
Posted by Pedro Giffuni <pf...@apache.org>.
----- Original Message -----
>
> On Sun, Aug 26, 2012 at 3:20 PM, Dave Fisher <da...@comcast.net>
> wrote:
>> Hi,
>>
>> We need to do more work to have proper compliance with Apache
> Infrastructure policy in managing external dependencies.
>>
>> I may not be precisely correct and am looking for confirmation, but In
> general i think we need to
>>
>> (1) Completely avoid using svn.apache.org. I don't think we are allowed
> to do this even as a backup URL.
>>
>> (2) Use mirrors or maven for ASF dependencies where we use the current
> release. If we use mirrors then archive.apache.org should be the backup for the
> mirror so that we aren't in trouble if the project has a release. If a maven
> repository were used then there would be no issue.
>>
>> (3) If we use mirrors then we should allow the user to choose which mirror.
>>
>> If we decide to take the time to go the maven route. I can use the example
> of ant and maven repos from the Apache POI build.xml.
>>
>> Notes about maven repos. Infra [1], maven central [2] and example of an
> externally hosted repo [3]
>>
>> This area needs careful attention.
>>
>
> Note that this move is exactly the wrong thing to do if we want have
> buildbots build binaries that are assumed to be safe and therefore
> signable. Instead of the security and verifiability of ASF-run host,
> we're putting the dependencies off to a dozen different remote sites,
> with no visibility into their site's mechanisms for vetting changes,
> access controls, auditability of changes, even basics like ensuring
> domain names are renewed and not poached by others.
>
This is not true. We use checksums precisely to certify the source tarballs
haven't been tampered with.
> Do we really think other websites are as secure as the ones that Infra
> operates? If so we should move the source code to the other sites as
> well, right?
>
Yes, upstream providers generate checksums and do proper
maintainance of their packages but perhaps more importantly by
working with them we can leverage their mirrors more efficiently.
I agree with Dave; we have to get rid of those tarballs
in SVN. Python 2.6.1 will be axed today, BTW.
I haven't looked at Maven but it sounds like something
we should look at..
Pedro.
Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
Posted by Rob Weir <ro...@apache.org>.
On Sun, Aug 26, 2012 at 3:20 PM, Dave Fisher <da...@comcast.net> wrote:
> Hi,
>
> We need to do more work to have proper compliance with Apache Infrastructure policy in managing external dependencies.
>
> I may not be precisely correct and am looking for confirmation, but In general i think we need to
>
> (1) Completely avoid using svn.apache.org. I don't think we are allowed to do this even as a backup URL.
>
> (2) Use mirrors or maven for ASF dependencies where we use the current release. If we use mirrors then archive.apache.org should be the backup for the mirror so that we aren't in trouble if the project has a release. If a maven repository were used then there would be no issue.
>
> (3) If we use mirrors then we should allow the user to choose which mirror.
>
> If we decide to take the time to go the maven route. I can use the example of ant and maven repos from the Apache POI build.xml.
>
> Notes about maven repos. Infra [1], maven central [2] and example of an externally hosted repo [3]
>
> This area needs careful attention.
>
Note that this move is exactly the wrong thing to do if we want have
buildbots build binaries that are assumed to be safe and therefore
signable. Instead of the security and verifiability of ASF-run host,
we're putting the dependencies off to a dozen different remote sites,
with no visibility into their site's mechanisms for vetting changes,
access controls, auditability of changes, even basics like ensuring
domain names are renewed and not poached by others.
Do we really think other websites are as secure as the ones that Infra
operates? If so we should move the source code to the other sites as
well, right?
No easy resolution of this, but we might mitigate the risk by putting
all of the dependencies to Apache-Extras and load from there
primarily. And if at all possible make sure all change notifications
from there get echoed to the ooo-committs lis. We have a better
chance of exercising now screwing up if we control rather than having
multiple 3rd parties control.
Another option would be to use cryptographic means to ensure the
integrity of the remote dependencies, e.g., detached signatures. That
doesn't protect us from another website going down, temporarily or
permanently, but it does allow us to verify that what we are
downloading has not been tampered with.
-Rob
> The current script is here: main/solenv/bin/download_external_dependencies.pl
>
> Regards,
> Dave
>
> [1] http://apache.org/dev/repository-faq.html and
> [2] http://maven.apache.org/guides/mini/guide-central-repository-upload.html
> [3] http://repo.maven.apache.org/maven2/javax/activation/activation/1.0.2/activation-1.0.2.pom
>
>
> On Aug 26, 2012, at 11:58 AM, wave@apache.org wrote:
>
>> Author: wave
>> Date: Sun Aug 26 18:58:08 2012
>> New Revision: 1377482
>>
>> URL: http://svn.apache.org/viewvc?rev=1377482&view=rev
>> Log:
>> one more small step to infra compliance. still to do removing use of svn as a backup and for current releases of ASF software the archive is not proper - either a mirror or the maven repository is required.
>>
>> Modified:
>> incubator/ooo/trunk/main/external_deps.lst
>>
>> Modified: incubator/ooo/trunk/main/external_deps.lst
>> URL: http://svn.apache.org/viewvc/incubator/ooo/trunk/main/external_deps.lst?rev=1377482&r1=1377481&r2=1377482&view=diff
>> ==============================================================================
>> --- incubator/ooo/trunk/main/external_deps.lst (original)
>> +++ incubator/ooo/trunk/main/external_deps.lst Sun Aug 26 18:58:08 2012
>> @@ -72,7 +72,7 @@ if ( true )
>> if (SOLAR_JAVA == TRUE)
>> MD5 = 17960f35b2239654ba608cf1f3e256b3
>> name = lucene-2.9.4-src.tar.gz
>> - URL1 = http://www.us.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
>> + URL1 = http://archive.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
>> URL2 = $(OOO_EXTRAS)$(MD5)-$(name)
>> # Fall back to a version in SVN from a previous revsion.
>> URL3 = http://svn.apache.org/repos/asf/!svn/bc/1337615/incubator/ooo/trunk/ext_sources/$(MD5)-$(name)
>>
>>
>
Re: svn commit: r1377482 - /incubator/ooo/trunk/main/external_deps.lst
Posted by Andre Fischer <aw...@gmail.com>.
On 26.08.2012 21:20, Dave Fisher wrote:
> Hi,
>
> We need to do more work to have proper compliance with Apache Infrastructure policy in managing external dependencies.
>
> I may not be precisely correct and am looking for confirmation, but In general i think we need to
>
> (1) Completely avoid using svn.apache.org. I don't think we are allowed to do this even as a backup URL.
Removing svn.apache.org was planned for after the release 3.4.1. I would
have done it this week. Thanks that you took care of it.
>
> (2) Use mirrors or maven for ASF dependencies where we use the current release. If we use mirrors then archive.apache.org should be the backup for the mirror so that we aren't in trouble if the project has a release. If a maven repository were used then there would be no issue.
Using ASF mirrors is difficult to do automatically. Doing the same for
projects hosted on SourceForge is easy. That is the reason why some ASF
dependencies are fetched from apache-extras.
Apache extras *is* the backup for all external dependencies that are not
extensions.
>
> (3) If we use mirrors then we should allow the user to choose which mirror.
That would break every automatic build.
But before we start making changes we should finally figure out the
policies that constrain our technical choices. I agree that the current
download mechanism is not perfect. One reason for that is that the
policies regarding licenses of the tarballs and possible download
locations for them are a moving target.
In the past months I was always trying to find a technical solution that
a) would work reliably
b) could be implemented in the short time until the next release and
c) would fit the newest requirements of where we were allowed to store
the tarballs.
If using the original servers is not the policy de jour anymore, fine.
If SHA1 is better than MD5, good. If maven is "better" than
apache-extras, excellent.
We should just make up our minds.
-Andre
>
> If we decide to take the time to go the maven route. I can use the example of ant and maven repos from the Apache POI build.xml.
>
> Notes about maven repos. Infra [1], maven central [2] and example of an externally hosted repo [3]
>
> This area needs careful attention.
>
> The current script is here: main/solenv/bin/download_external_dependencies.pl
>
> Regards,
> Dave
>
> [1] http://apache.org/dev/repository-faq.html and
> [2] http://maven.apache.org/guides/mini/guide-central-repository-upload.html
> [3] http://repo.maven.apache.org/maven2/javax/activation/activation/1.0.2/activation-1.0.2.pom
>
>
> On Aug 26, 2012, at 11:58 AM, wave@apache.org wrote:
>
>> Author: wave
>> Date: Sun Aug 26 18:58:08 2012
>> New Revision: 1377482
>>
>> URL: http://svn.apache.org/viewvc?rev=1377482&view=rev
>> Log:
>> one more small step to infra compliance. still to do removing use of svn as a backup and for current releases of ASF software the archive is not proper - either a mirror or the maven repository is required.
>>
>> Modified:
>> incubator/ooo/trunk/main/external_deps.lst
>>
>> Modified: incubator/ooo/trunk/main/external_deps.lst
>> URL: http://svn.apache.org/viewvc/incubator/ooo/trunk/main/external_deps.lst?rev=1377482&r1=1377481&r2=1377482&view=diff
>> ==============================================================================
>> --- incubator/ooo/trunk/main/external_deps.lst (original)
>> +++ incubator/ooo/trunk/main/external_deps.lst Sun Aug 26 18:58:08 2012
>> @@ -72,7 +72,7 @@ if ( true )
>> if (SOLAR_JAVA == TRUE)
>> MD5 = 17960f35b2239654ba608cf1f3e256b3
>> name = lucene-2.9.4-src.tar.gz
>> - URL1 = http://www.us.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
>> + URL1 = http://archive.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz
>> URL2 = $(OOO_EXTRAS)$(MD5)-$(name)
>> # Fall back to a version in SVN from a previous revsion.
>> URL3 = http://svn.apache.org/repos/asf/!svn/bc/1337615/incubator/ooo/trunk/ext_sources/$(MD5)-$(name)
>>
>>
>