You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@netbeans.apache.org by Antonio <an...@vieiro.net> on 2017/11/01 09:29:13 UTC
Do we want multiple download servers
Hi all,
PROBLEM:
The mylyn binaries are not available in Maven central. Nor in any other
Maven repository.
SOLUTION:
We could use the eclipse downloads system, with automatic mirror
selection, with [1].
So, for instance, to download the jar file
org.eclipse.mylyn.tasks.core_3.17.0.v20150828-2026.jar for the
o.eclipse.mylyn.tasks.core module we could use the url [2], and we could
then use an appropriate SHA-1 to verify that the download is correct.
All mylyn binaries are available there (maybe other binaries as well).
Currently the build system uses a single server
(nbbuild/default-properties.xml|binaries.server). But we could specify
multiple download servers and try to download from them in order, until
the download succeeds.
QUESTIONS:
Would it be a good idea to add support for multiple URLs into
"binaries.server"? Is it ok to download binaries from the
www.eclipse.org/downloads URL? Do you want me to submit a PR with this?
Thanks,
Antonio
[1]
http://www.eclipse.org/downloads/download.php?file=/mylyn/drops/3.17.0/v20150909-1855/plugins
[2]
http://www.eclipse.org/downloads/download.php?file=/mylyn/drops/3.17.0/v20150909-1855/plugins/org.eclipse.mylyn.tasks.core_3.17.0.v20150828-2026.jar
Re: Do we want multiple download servers
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Wed, Nov 1, 2017 at 3:12 PM, Emilian Bold <em...@protonmail.ch> wrote:
> ...Since Apache does provide "convenience" binaries, ie. it does serve dependencies,
> how is it suddenly strange to have a place to store only the dependencies?...
It's different if those binaries are from Apache projects or third parties.
Also, starting to be another Maven Central has costs and legal
implications, we cannot just do it without asking our infrastructure
team.
The current ASF mechanism for distributing *our* binaries and releases
is https://www.apache.org/mirrors/
-Bertrand
Re: Do we want multiple download servers
Posted by Emilian Bold <em...@protonmail.ch>.
You know, in the old days when you just released a .tar.gz it was perfectly acceptable to have a lib/ folder in the archive with your dependencies.
Because there was no Maven Central or whatever and doing wget from 10 different FTP server for all the dependencies was seen as inconvenient.
Since Apache does provide "convenience" binaries, ie. it does serve dependencies, how is it suddenly strange to have a place to store only the dependencies?
In these modern times what we actually need is Git LFS and store those libraries there https://git-lfs.github.com
--emi
>-------- Original Message --------
>Subject: Re: Do we want multiple download servers
>Local Time: November 1, 2017 4:01 PM
>UTC Time: November 1, 2017 2:01 PM
>From: bdelacretaz@apache.org
>To: dev@netbeans.incubator.apache.org
>
>On Wed, Nov 1, 2017 at 2:57 PM, Emilian Bold emilian.bold@protonmail.ch wrote:
>>...is there any reason INFRA could not provide us a basic FTP account to carry on http://hg.netbeans.org/binaries under Apache?...
>>
>> It depends on the cost, and hosting third-party binaries that we do
>> not control might be a problem.
>>
>> But maybe not - best is to write a concise problem description that we
>> can discuss here and maybe forward to INFRA.
>>
>> -Bertrand
>
Re: Do we want multiple download servers
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Wed, Nov 1, 2017 at 2:57 PM, Emilian Bold <em...@protonmail.ch> wrote:
> ...is there any reason INFRA could not provide us a basic FTP account to carry on http://hg.netbeans.org/binaries under Apache?...
It depends on the cost, and hosting third-party binaries that we do
not control might be a problem.
But maybe not - best is to write a concise problem description that we
can discuss here and maybe forward to INFRA.
-Bertrand
Re: Do we want multiple download servers
Posted by Emilian Bold <em...@protonmail.ch>.
Actually, is there any reason INFRA could not provide us a basic FTP account to carry on http://hg.netbeans.org/binaries under Apache?
Then we wouldn't have to use OSUOSL, which are also an Apache mirror provider, so they already have a separate deal with the ASF.
This whole setup is just a way to keep the git repo size under control.
Because alternatively we could just commit 3rd party JARs, considering they *are* an IP cleared project dependency.
--emi
>-------- Original Message --------
>Subject: Re: Do we want multiple download servers
>Local Time: November 1, 2017 3:32 PM
>UTC Time: November 1, 2017 1:32 PM
>From: bdelacretaz@apache.org
>To: dev@netbeans.incubator.apache.org
>
>Hi,
>
> On Wed, Nov 1, 2017 at 2:27 PM, Emilian Bold emilian.bold@protonmail.ch wrote:
>>...In the future http://hg.netbeans.org/binaries might be going away, but we still have http://netbeans.osuosl.org/binaries/
>> which is under my control...
>>
>> Note that any such resources should be put under control of the
>> NetBeans PPMC, for bus factor reasons.
>>
>> Also, before graduating, a list of non-Apache resource on which the
>> project depends should be established.
>>
>> -Bertrand
Re: Do we want multiple download servers
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi,
On Wed, Nov 1, 2017 at 2:27 PM, Emilian Bold <em...@protonmail.ch> wrote:
> ...In the future http://hg.netbeans.org/binaries might be going away, but we still have http://netbeans.osuosl.org/binaries/
> which is under my control...
Note that any such resources should be put under control of the
NetBeans PPMC, for bus factor reasons.
Also, before graduating, a list of non-Apache resource on which the
project depends should be established.
-Bertrand
Re: Do we want multiple download servers
Posted by Emilian Bold <em...@protonmail.ch>.
> I know this is a problem of mine, worring about these things, so I apologize for being so tiresome asking questions all the time. I’ll focus on finishing the licensing stuff and leave the final solution to this for later.
This is exactly the kind of questions that need to be asked. No need to apologize, you are doing great work!
--emi
On mie., nov. 1, 2017 at 9:39 p.m., Antonio Vieiro <an...@vieiro.net> wrote:
> Hi all, I know I’ve been raising lots of questions about binary dependencies lately. The fact is that I don’t feel comfortable having them stored in hg.netbeans.org/binaries knowing this is going away. I’d prefer having as a solid build system as we can get. I’d prefer not having to upgrade/downgrade versions because artifacts are not in Maven central. Currently we have 10 o.eclipse.mylyn.* modules that depend on binaries that are not in Maven central, that’s why I’m searching alternatives to hg.netbeans.org/binaries. I’m glad to hear Emilian has an alternative repo. I know this is a problem of mine, worring about these things, so I apologize for being so tiresome asking questions all the time. I’ll focus on finishing the licensing stuff and leave the final solution to this for later. Anyway I’m glad there’s been some discussion on the matter. Cheers, Antonio > El 1 nov 2017, a las 14:27, Emilian Bold escribió: > > If we cannot use Maven Central or some other "big" official repositories that guarantee some integrity, then we might as well use our own setup. > > In the future http://hg.netbeans.org/binaries might be going away, but we still have http://netbeans.osuosl.org/binaries/ which is under my control and where we have a lot of bandwidth and enough space. Also see the article about this https://jaxenter.com/netbeans/towards-building-netbeans-new-home > > So I suggest we don't make this overly complicated. Right now we can leave some binaries as they are and keep using http://hg.netbeans.org/binaries until it goes down. Later on we can switch to http://netbeans.osuosl.org/binaries/ (and perhaps also add a new folder just for new binaries, perhaps http://netbeans.osuosl.org/apache-binaries/ or something). > > --emi > >> -------- Original Message -------- >> Subject: Do we want multiple download servers >> Local Time: November 1, 2017 11:29 AM >> UTC Time: November 1, 2017 9:29 AM >> From: antonio@vieiro.net >> To: dev@netbeans.incubator.apache.org >> >> Hi all, >> >> PROBLEM: >> >> The mylyn binaries are not available in Maven central. Nor in any other >> Maven repository. >> >> SOLUTION: >> >> We could use the eclipse downloads system, with automatic mirror >> selection, with [1]. >> >> So, for instance, to download the jar file >> org.eclipse.mylyn.tasks.core_3.17.0.v20150828-2026.jar for the >> o.eclipse.mylyn.tasks.core module we could use the url [2], and we could >> then use an appropriate SHA-1 to verify that the download is correct. >> >> All mylyn binaries are available there (maybe other binaries as well). >> >> Currently the build system uses a single server >> (nbbuild/default-properties.xml|binaries.server). But we could specify >> multiple download servers and try to download from them in order, until >> the download succeeds. >> >> QUESTIONS: >> >> Would it be a good idea to add support for multiple URLs into >> "binaries.server"? Is it ok to download binaries from the >> www.eclipse.org/downloads URL? Do you want me to submit a PR with this? >> >> Thanks, >> Antonio >> >> >> [1] >> http://www.eclipse.org/downloads/download.php?file=/mylyn/drops/3.17.0/v20150909-1855/plugins >> >> [2] >> http://www.eclipse.org/downloads/download.php?file=/mylyn/drops/3.17.0/v20150909-1855/plugins/org.eclipse.mylyn.tasks.core_3.17.0.v20150828-2026.jar @protonmail.ch>
Re: Do we want multiple download servers
Posted by Antonio Vieiro <an...@vieiro.net>.
Hi all,
I know I’ve been raising lots of questions about binary dependencies lately. The fact is that I don’t feel comfortable having them stored in hg.netbeans.org/binaries knowing this is going away. I’d prefer having as a solid build system as we can get. I’d prefer not having to upgrade/downgrade versions because artifacts are not in Maven central.
Currently we have 10 o.eclipse.mylyn.* modules that depend on binaries that are not in Maven central, that’s why I’m searching alternatives to hg.netbeans.org/binaries. I’m glad to hear Emilian has an alternative repo.
I know this is a problem of mine, worring about these things, so I apologize for being so tiresome asking questions all the time. I’ll focus on finishing the licensing stuff and leave the final solution to this for later.
Anyway I’m glad there’s been some discussion on the matter.
Cheers,
Antonio
> El 1 nov 2017, a las 14:27, Emilian Bold <em...@protonmail.ch> escribió:
>
> If we cannot use Maven Central or some other "big" official repositories that guarantee some integrity, then we might as well use our own setup.
>
> In the future http://hg.netbeans.org/binaries might be going away, but we still have http://netbeans.osuosl.org/binaries/ which is under my control and where we have a lot of bandwidth and enough space. Also see the article about this https://jaxenter.com/netbeans/towards-building-netbeans-new-home
>
> So I suggest we don't make this overly complicated. Right now we can leave some binaries as they are and keep using http://hg.netbeans.org/binaries until it goes down. Later on we can switch to http://netbeans.osuosl.org/binaries/ (and perhaps also add a new folder just for new binaries, perhaps http://netbeans.osuosl.org/apache-binaries/ or something).
>
> --emi
>
>> -------- Original Message --------
>> Subject: Do we want multiple download servers
>> Local Time: November 1, 2017 11:29 AM
>> UTC Time: November 1, 2017 9:29 AM
>> From: antonio@vieiro.net
>> To: dev@netbeans.incubator.apache.org
>>
>> Hi all,
>>
>> PROBLEM:
>>
>> The mylyn binaries are not available in Maven central. Nor in any other
>> Maven repository.
>>
>> SOLUTION:
>>
>> We could use the eclipse downloads system, with automatic mirror
>> selection, with [1].
>>
>> So, for instance, to download the jar file
>> org.eclipse.mylyn.tasks.core_3.17.0.v20150828-2026.jar for the
>> o.eclipse.mylyn.tasks.core module we could use the url [2], and we could
>> then use an appropriate SHA-1 to verify that the download is correct.
>>
>> All mylyn binaries are available there (maybe other binaries as well).
>>
>> Currently the build system uses a single server
>> (nbbuild/default-properties.xml|binaries.server). But we could specify
>> multiple download servers and try to download from them in order, until
>> the download succeeds.
>>
>> QUESTIONS:
>>
>> Would it be a good idea to add support for multiple URLs into
>> "binaries.server"? Is it ok to download binaries from the
>> www.eclipse.org/downloads URL? Do you want me to submit a PR with this?
>>
>> Thanks,
>> Antonio
>>
>>
>> [1]
>> http://www.eclipse.org/downloads/download.php?file=/mylyn/drops/3.17.0/v20150909-1855/plugins
>>
>> [2]
>> http://www.eclipse.org/downloads/download.php?file=/mylyn/drops/3.17.0/v20150909-1855/plugins/org.eclipse.mylyn.tasks.core_3.17.0.v20150828-2026.jar
Re: Do we want multiple download servers
Posted by Emilian Bold <em...@protonmail.ch>.
If we cannot use Maven Central or some other "big" official repositories that guarantee some integrity, then we might as well use our own setup.
In the future http://hg.netbeans.org/binaries might be going away, but we still have http://netbeans.osuosl.org/binaries/ which is under my control and where we have a lot of bandwidth and enough space. Also see the article about this https://jaxenter.com/netbeans/towards-building-netbeans-new-home
So I suggest we don't make this overly complicated. Right now we can leave some binaries as they are and keep using http://hg.netbeans.org/binaries until it goes down. Later on we can switch to http://netbeans.osuosl.org/binaries/ (and perhaps also add a new folder just for new binaries, perhaps http://netbeans.osuosl.org/apache-binaries/ or something).
--emi
>-------- Original Message --------
>Subject: Do we want multiple download servers
>Local Time: November 1, 2017 11:29 AM
>UTC Time: November 1, 2017 9:29 AM
>From: antonio@vieiro.net
>To: dev@netbeans.incubator.apache.org
>
>Hi all,
>
> PROBLEM:
>
> The mylyn binaries are not available in Maven central. Nor in any other
> Maven repository.
>
> SOLUTION:
>
> We could use the eclipse downloads system, with automatic mirror
> selection, with [1].
>
> So, for instance, to download the jar file
> org.eclipse.mylyn.tasks.core_3.17.0.v20150828-2026.jar for the
> o.eclipse.mylyn.tasks.core module we could use the url [2], and we could
> then use an appropriate SHA-1 to verify that the download is correct.
>
> All mylyn binaries are available there (maybe other binaries as well).
>
> Currently the build system uses a single server
> (nbbuild/default-properties.xml|binaries.server). But we could specify
> multiple download servers and try to download from them in order, until
> the download succeeds.
>
> QUESTIONS:
>
> Would it be a good idea to add support for multiple URLs into
> "binaries.server"? Is it ok to download binaries from the
>www.eclipse.org/downloads URL? Do you want me to submit a PR with this?
>
> Thanks,
> Antonio
>
>
> [1]
>http://www.eclipse.org/downloads/download.php?file=/mylyn/drops/3.17.0/v20150909-1855/plugins
>
> [2]
>http://www.eclipse.org/downloads/download.php?file=/mylyn/drops/3.17.0/v20150909-1855/plugins/org.eclipse.mylyn.tasks.core_3.17.0.v20150828-2026.jar
>
Re: Do we want multiple download servers
Posted by Ate Douma <at...@douma.nu>.
On 2017-11-01 17:29, Matthias Bläsing wrote:
> Hey,
>
> the big question - are there alternatives? How well does git cope with
> binaries and are there experiences with big repositories?
>
> The JNA project decided, that holding the released binaries in the git
> history would be a good idea. The library itself is a few megabytes,
> the repository currently takes nearly 500 MB.
>
> If netbeans begins to regularly update binaries I fear, that the
> repository grows into unhandleable sizes.
>
> What about this:
>
> Instead of maven coordinates full URLs are specified in binaries-list.
> As long as the hash is checked, this should still be secure.
>
> This way also sources for binaries, not backed by maven (eclipse) could
> be supported.
>
> If we ensure that only stable URLs are used (Maven, Eclipse, W3C) we
> would be save - or we establish a separate repository for the binaries,
> as Emilian suggested.
>
> For now I think I found most eclipse binaries on maven central, so we
> could focus on preparing a release for now.
If most Eclipse binaries already are available from maven central, maybe
we can ask *them* to also upload/push these missing binaries there?
Sometimes things like these can be resolved by just asking friendly :-)
Ate
>
> Greetings
>
> Matthias
>
>
> Am Mittwoch, den 01.11.2017, 10:29 +0100 schrieb Antonio:
>> Hi all,
>>
>> PROBLEM:
>>
>> The mylyn binaries are not available in Maven central. Nor in any
>> other
>> Maven repository.
>>
>> SOLUTION:
>>
>> We could use the eclipse downloads system, with automatic mirror
>> selection, with [1].
>>
>> So, for instance, to download the jar file
>> org.eclipse.mylyn.tasks.core_3.17.0.v20150828-2026.jar for the
>> o.eclipse.mylyn.tasks.core module we could use the url [2], and we
>> could
>> then use an appropriate SHA-1 to verify that the download is correct.
>>
>> All mylyn binaries are available there (maybe other binaries as
>> well).
>>
>> Currently the build system uses a single server
>> (nbbuild/default-properties.xml|binaries.server). But we could
>> specify
>> multiple download servers and try to download from them in order,
>> until
>> the download succeeds.
>>
>> QUESTIONS:
>>
>> Would it be a good idea to add support for multiple URLs into
>> "binaries.server"? Is it ok to download binaries from the
>> www.eclipse.org/downloads URL? Do you want me to submit a PR with
>> this?
>>
>> Thanks,
>> Antonio
>>
>>
>> [1]
>> http://www.eclipse.org/downloads/download.php?file=/mylyn/drops/3.17.
>> 0/v20150909-1855/plugins
>>
>> [2]
>> http://www.eclipse.org/downloads/download.php?file=/mylyn/drops/3.17.
>> 0/v20150909-
>> 1855/plugins/org.eclipse.mylyn.tasks.core_3.17.0.v20150828-2026.jar
Re: Do we want multiple download servers
Posted by Matthias Bläsing <mb...@doppel-helix.eu>.
Hey,
the big question - are there alternatives? How well does git cope with
binaries and are there experiences with big repositories?
The JNA project decided, that holding the released binaries in the git
history would be a good idea. The library itself is a few megabytes,
the repository currently takes nearly 500 MB.
If netbeans begins to regularly update binaries I fear, that the
repository grows into unhandleable sizes.
What about this:
Instead of maven coordinates full URLs are specified in binaries-list.
As long as the hash is checked, this should still be secure.
This way also sources for binaries, not backed by maven (eclipse) could
be supported.
If we ensure that only stable URLs are used (Maven, Eclipse, W3C) we
would be save - or we establish a separate repository for the binaries,
as Emilian suggested.
For now I think I found most eclipse binaries on maven central, so we
could focus on preparing a release for now.
Greetings
Matthias
Am Mittwoch, den 01.11.2017, 10:29 +0100 schrieb Antonio:
> Hi all,
>
> PROBLEM:
>
> The mylyn binaries are not available in Maven central. Nor in any
> other
> Maven repository.
>
> SOLUTION:
>
> We could use the eclipse downloads system, with automatic mirror
> selection, with [1].
>
> So, for instance, to download the jar file
> org.eclipse.mylyn.tasks.core_3.17.0.v20150828-2026.jar for the
> o.eclipse.mylyn.tasks.core module we could use the url [2], and we
> could
> then use an appropriate SHA-1 to verify that the download is correct.
>
> All mylyn binaries are available there (maybe other binaries as
> well).
>
> Currently the build system uses a single server
> (nbbuild/default-properties.xml|binaries.server). But we could
> specify
> multiple download servers and try to download from them in order,
> until
> the download succeeds.
>
> QUESTIONS:
>
> Would it be a good idea to add support for multiple URLs into
> "binaries.server"? Is it ok to download binaries from the
> www.eclipse.org/downloads URL? Do you want me to submit a PR with
> this?
>
> Thanks,
> Antonio
>
>
> [1]
> http://www.eclipse.org/downloads/download.php?file=/mylyn/drops/3.17.
> 0/v20150909-1855/plugins
>
> [2]
> http://www.eclipse.org/downloads/download.php?file=/mylyn/drops/3.17.
> 0/v20150909-
> 1855/plugins/org.eclipse.mylyn.tasks.core_3.17.0.v20150828-2026.jar