You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@netbeans.apache.org by Antonio <an...@vieiro.net> on 2018/04/11 06:39:12 UTC
Summary of How/where exactly to host the NBMs for the 9.0 release
Hi all,
In order to understand this long thread (about NETBEANS-330 [1]) I tried
to summarize it. Please review and send corrections & questions as
appropriate.
Kind regards,
Antonio
== Objectives
Host the NetBeans 9.0 Update Center on Apache infrastructure.
== Constraints/facts/options
- We cannot host it on a website due to bandwidth requirements of 3-5
Tb/month.
- We must use the Apache Mirror system and their "closer.cgi,
closer.lua" cgi scripts [2] to select the closest mirror to the user.
This script can either redirect directly to the closest binary file
(ready to download), or return a JSON response with a list of closest
mirrors.
- Distributing through the Apache Mirror system requires a proper
release, with voting, approval and signing, etc.
- Geertjan says that prior to the final release we could do an rc
release to have this feature tested.
- The UC catalog xml file can be hosted in the mirror system. For this
to work the catalog xml file must contain relative paths, so when aged
releases are moved away from the mirror system and into the archive
things keep working.
- We can HTTP redirect to the proper catalog.xml file...
a) ... From our website, redirecting to the Apache mirror system or to
the archive with an .htaccess file (under our control).
b) ... Idem, by asking infra to modify our server configuration (more
performant but requires Infra tickets).
c) ... Using a script of ours hosted at the "VM" (a web server ours
currently hosting selenium), that may also track/log some statistics.
- The NetBeans UC module will make a request for the XML file to our
website, from there will be redirected to the mirror/archive. Later on
it will build up proper URLs (using some prefix and the relative paths
inside the catalog xml file) for the final NBM downloads.
- Mirror download statistics may be currently downloaded from
https://www.apache.org/dyn/stats/netbeans.log
- Web server statistics are available through a very detailed request to
infra. They say they're not in the "counting business" :-)
- Jan is working in NBM generation (with NOTICE & LICENSE, etc.).
Jenkins now generates these artifacts:
https://builds.apache.org/view/Incubator%20Projects/job/incubator-netbeans-linux/lastSuccessfulBuild/artifact/nbbuild/nbms/
== Open issues & questions
1- It seems we won't be able to host some OSGi bundles in the mirror
system due to licensing issues?
2- Mirrors are not as reliable as a CDN: files can be corrupt, or
whatever. The UC module should verify the integrity of the downloaded
files, etc.
3- We may want to select a small set of NBM files for the first run, to
workaround previous licensing problems.
[1]
https://issues.apache.org/jira/browse/NETBEANS-330
[2]
https://reference.apache.org/pmc/mirror_scripts
On 05/04/18 14:38, Geertjan Wielenga wrote:
> Hi all,
>
> We need to nail down this one and I think the key blocker is that there are
> different ideas about what this is about:
>
> https://issues.apache.org/jira/browse/NETBEANS-330
>
> The above is not about the Plugin Portal.
>
> If I understand it correctly, this is about where the NBMs (which ones? how
> many? do we know?) and the related XML file (a.k.a. update center) will be
> hosted.
>
> AFAIK, the XML file and the NBMs could be put onto our Apache NetBeans VM
> just like Synergy:
>
> http://netbeans-vm.apache.org/synergy
>
> The key question remains, which NBMs are we talking about here, applicable
> to the 9.0 release, I think.
>
> Thanks,
>
> Gj
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Resurrect: Summary of How/where exactly to host the NBMs for the
9.0 release
Posted by Laszlo Kishalmi <la...@gmail.com>.
Changed the rewrite rule on netbeans-vm to forward to the Apache
Archives for 9.0 nbm-s.
Also archived the 9.0 release.
On 12/27/18 5:38 AM, Antonio wrote:
> Gavin McDonald says they'll tell us if there're any traffic problems,
> and that 9.0 is in the archive, and that we should use that.
>
> http://archive.apache.org/dist/incubator/netbeans/incubating-netbeans-java/incubating-9.0/
>
>
> Cheers,
> Antonio
>
> El 27/12/2018 a las 14:26, Antonio escribió:
>> Hi all,
>>
>> I'm asking at infra.chat if we should use the archive or
>> netbeans-vm.apache.org.
>>
>> Cheers,
>> Antonio
>>
>>
>> El 27/12/2018 a las 9:46, Emilian Bold escribió:
>>>> That's no questions. My only concern is where to store.
>>>
>>> We store on Apache infra and if it's too much for them we ask them for
>>> suggestions or new servers.
>>>
>>> I understand how you computed but for older nbms I don't believe we
>>> will have even a fraction of those maximum downloads.
>>>
>>> --emi
>>>
>>> http://coolbeans.xyz/ - CoolBeans: An IDE for Java, JavaEE, PHP and
>>> more!
>>>
>>> On Thu, Dec 27, 2018 at 10:44 AM Laszlo Kishalmi
>>> <la...@gmail.com> wrote:
>>>>
>>>> On 12/26/18 11:23 PM, Emilian Bold wrote:
>>>>> Considering the are no updates, why would we even serve the nbms? To
>>>>> older 8.2 installs or what? Won't those older installs request the 10
>>>>> nbms anyhow? So, I don't see how those nbms would get so many
>>>>> downloads.
>>>>
>>>> I've just really calculated the worst case scenario for the download
>>>> traffic:
>>>>
>>>> 1. Number of request per month on updates.xml.gz is 22 - 25
>>>> thousand I
>>>> took 25000
>>>> 2. NetBeans 10.0 nbm-s take more space (118Mb) than 9.0, so I
>>>> calculated with 120Mb
>>>>
>>>> So if everyone would download the whole nbm structure whenever it hits
>>>> the updates.xml.gz, then we could have 25000 * 120 Mb = 3 Tb per month
>>>> download traffic.
>>>>
>>>>> We can put the smaller updates.xml.gz anywhere, it won't make any
>>>>> impact on bandwidth.
>>>>>
>>>>> I would not use any 3rd party solution unless Apache infra tell us
>>>>> so.
>>>>> The one thing Apache provides to projects is infrastructure.
>>>> Well, Apache provides more than infra, however that infra not
>>>> necessary
>>>> designed for a project like us., that are producing 650+ artifacts,
>>>> weighs ~500 Mb with every release.
>>>>>
>>>>> I say we pick the simplest solution: use archive.apache.org for
>>>>> everything. It will serve only the updates.xml.gz in reality and if
>>>>> they do get too much traffic infra will let us know and we figure
>>>>> something out then.
>>>> Well, that's might be the easiest solution, still option 2 or 3
>>>> would be
>>>> easy to implement as well and clean up our project layout on
>>>> dist.apache.org.
>>>>>
>>>>> BTW, I still believe we have to create, sign and store the nbms, even
>>>>> if they are only convenience binaries.
>>>> That's no questions. My only concern is where to store.
>>>>>
>>>>> --emi
>>>>>
>>>>> http://coolbeans.xyz/ - CoolBeans: An IDE for Java, JavaEE, PHP
>>>>> and more!
>>>>>
>>>>> On Thu, Dec 27, 2018 at 8:11 AM Laszlo Kishalmi
>>>>> <la...@gmail.com> wrote:
>>>>>> Dear all,
>>>>>>
>>>>>> I have to bring up this old item again. According to the Apache
>>>>>> release
>>>>>> processes, I'd need to archive the current 9.0 release from the
>>>>>> site.
>>>>>> this means that our 9.0 NBM-s are going to be removed from there as
>>>>>> well. The files are going to be available on archive.apache.org
>>>>>> which is
>>>>>> not mirrored. Only the most recent release are supposed to be out
>>>>>> on the
>>>>>> release and their mirrors.
>>>>>>
>>>>>> This means we need to find some place to our 9.0 nbms. I also
>>>>>> won't mind
>>>>>> if we could move away storing our nbms, "officially" (there are no
>>>>>> official binary releases, yet) on some other place than Apache dist
>>>>>> infrastructure. It's a bit weird to sign and upload 650+ binaries
>>>>>> with
>>>>>> each and every release. So let's go back to square one.
>>>>>>
>>>>>> 1. We can change the redirection to the Apache archive site.
>>>>>> That is
>>>>>> not mirrored, and initially we would put a real traffic on
>>>>>> there.
>>>>>> 2. We can serve the nbm-s from our netbeans-vm.
>>>>>> Checked the stats we get ~25000 requests a month on
>>>>>> updates.xml.gz
>>>>>> if we'd transfer all the update center (~120 Mb, for
>>>>>> NetBeans 10.0)
>>>>>> all the modules with each request for updates.xml.gz our
>>>>>> transfer
>>>>>> would be ~3 Tb. So we would definitely fit into the 3-5
>>>>>> Tb/per month
>>>>>> usage limit. That would be actually much less, as this is a
>>>>>> worst
>>>>>> case scenario.
>>>>>> 3. Search for a free third party solution.
>>>>>> I'd try Bintray, we can get 10 Gb space and CDN, so it is
>>>>>> pretty
>>>>>> reliable, the only concern is the 1 Tb of network limit per
>>>>>> month.
>>>>>> Unfortunately I only can guess that the 3 Tb is really
>>>>>> exaggerated.
>>>>>> All the log files I've found do not include the actual
>>>>>> download
>>>>>> statistics of our nbm-s
>>>>>> 4. Something else?
>>>>>>
>>>>>> I can't archive the current 9.0 till this thing is decided.
>>>>>>
>>>>>> On 4/10/18 11:39 PM, Antonio wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> In order to understand this long thread (about NETBEANS-330 [1]) I
>>>>>>> tried to summarize it. Please review and send corrections &
>>>>>>> questions
>>>>>>> as appropriate.
>>>>>>>
>>>>>>> Kind regards,
>>>>>>> Antonio
>>>>>>>
>>>>>>> == Objectives
>>>>>>>
>>>>>>> Host the NetBeans 9.0 Update Center on Apache infrastructure.
>>>>>>>
>>>>>>> == Constraints/facts/options
>>>>>>>
>>>>>>> - We cannot host it on a website due to bandwidth requirements
>>>>>>> of 3-5
>>>>>>> Tb/month.
>>>>>>>
>>>>>>> - We must use the Apache Mirror system and their "closer.cgi,
>>>>>>> closer.lua" cgi scripts [2] to select the closest mirror to the
>>>>>>> user.
>>>>>>> This script can either redirect directly to the closest binary file
>>>>>>> (ready to download), or return a JSON response with a list of
>>>>>>> closest
>>>>>>> mirrors.
>>>>>>>
>>>>>>> - Distributing through the Apache Mirror system requires a proper
>>>>>>> release, with voting, approval and signing, etc.
>>>>>>>
>>>>>>> - Geertjan says that prior to the final release we could do an rc
>>>>>>> release to have this feature tested.
>>>>>>>
>>>>>>> - The UC catalog xml file can be hosted in the mirror system.
>>>>>>> For this
>>>>>>> to work the catalog xml file must contain relative paths, so
>>>>>>> when aged
>>>>>>> releases are moved away from the mirror system and into the archive
>>>>>>> things keep working.
>>>>>>>
>>>>>>> - We can HTTP redirect to the proper catalog.xml file...
>>>>>>>
>>>>>>> a) ... From our website, redirecting to the Apache mirror system
>>>>>>> or to
>>>>>>> the archive with an .htaccess file (under our control).
>>>>>>>
>>>>>>> b) ... Idem, by asking infra to modify our server configuration
>>>>>>> (more
>>>>>>> performant but requires Infra tickets).
>>>>>>>
>>>>>>> c) ... Using a script of ours hosted at the "VM" (a web server ours
>>>>>>> currently hosting selenium), that may also track/log some
>>>>>>> statistics.
>>>>>>>
>>>>>>> - The NetBeans UC module will make a request for the XML file to
>>>>>>> our
>>>>>>> website, from there will be redirected to the mirror/archive.
>>>>>>> Later on
>>>>>>> it will build up proper URLs (using some prefix and the relative
>>>>>>> paths
>>>>>>> inside the catalog xml file) for the final NBM downloads.
>>>>>>>
>>>>>>> - Mirror download statistics may be currently downloaded from
>>>>>>> https://www.apache.org/dyn/stats/netbeans.log
>>>>>>>
>>>>>>> - Web server statistics are available through a very detailed
>>>>>>> request
>>>>>>> to infra. They say they're not in the "counting business" :-)
>>>>>>>
>>>>>>> - Jan is working in NBM generation (with NOTICE & LICENSE, etc.).
>>>>>>> Jenkins now generates these artifacts:
>>>>>>> https://builds.apache.org/view/Incubator%20Projects/job/incubator-netbeans-linux/lastSuccessfulBuild/artifact/nbbuild/nbms/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> == Open issues & questions
>>>>>>>
>>>>>>> 1- It seems we won't be able to host some OSGi bundles in the
>>>>>>> mirror
>>>>>>> system due to licensing issues?
>>>>>>>
>>>>>>> 2- Mirrors are not as reliable as a CDN: files can be corrupt, or
>>>>>>> whatever. The UC module should verify the integrity of the
>>>>>>> downloaded
>>>>>>> files, etc.
>>>>>>>
>>>>>>> 3- We may want to select a small set of NBM files for the first
>>>>>>> run,
>>>>>>> to workaround previous licensing problems.
>>>>>>>
>>>>>>>
>>>>>>> [1]
>>>>>>> https://issues.apache.org/jira/browse/NETBEANS-330
>>>>>>>
>>>>>>> [2]
>>>>>>> https://reference.apache.org/pmc/mirror_scripts
>>>>>>>
>>>>>>>
>>>>>>> On 05/04/18 14:38, Geertjan Wielenga wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> We need to nail down this one and I think the key blocker is that
>>>>>>>> there are
>>>>>>>> different ideas about what this is about:
>>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/NETBEANS-330
>>>>>>>>
>>>>>>>> The above is not about the Plugin Portal.
>>>>>>>>
>>>>>>>> If I understand it correctly, this is about where the NBMs (which
>>>>>>>> ones? how
>>>>>>>> many? do we know?) and the related XML file (a.k.a. update center)
>>>>>>>> will be
>>>>>>>> hosted.
>>>>>>>>
>>>>>>>> AFAIK, the XML file and the NBMs could be put onto our Apache
>>>>>>>> NetBeans VM
>>>>>>>> just like Synergy:
>>>>>>>>
>>>>>>>> http://netbeans-vm.apache.org/synergy
>>>>>>>>
>>>>>>>> The key question remains, which NBMs are we talking about here,
>>>>>>>> applicable
>>>>>>>> to the 9.0 release, I think.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Gj
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>>
>>>>>>> To unsubscribe, e-mail:
>>>>>>> dev-unsubscribe@netbeans.incubator.apache.org
>>>>>>> For additional commands, e-mail:
>>>>>>> dev-help@netbeans.incubator.apache.org
>>>>>>>
>>>>>>> For further information about the NetBeans mailing lists, visit:
>>>>>>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
>>>>> For additional commands, e-mail:
>>>>> dev-help@netbeans.incubator.apache.org
>>>>>
>>>>> For further information about the NetBeans mailing lists, visit:
>>>>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>>>>>
>>>>>
>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
>>> For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
>>>
>>> For further information about the NetBeans mailing lists, visit:
>>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
> For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Resurrect: Summary of How/where exactly to host the NBMs for the
9.0 release
Posted by Antonio <an...@vieiro.net>.
Gavin McDonald says they'll tell us if there're any traffic problems,
and that 9.0 is in the archive, and that we should use that.
http://archive.apache.org/dist/incubator/netbeans/incubating-netbeans-java/incubating-9.0/
Cheers,
Antonio
El 27/12/2018 a las 14:26, Antonio escribió:
> Hi all,
>
> I'm asking at infra.chat if we should use the archive or
> netbeans-vm.apache.org.
>
> Cheers,
> Antonio
>
>
> El 27/12/2018 a las 9:46, Emilian Bold escribió:
>>> That's no questions. My only concern is where to store.
>>
>> We store on Apache infra and if it's too much for them we ask them for
>> suggestions or new servers.
>>
>> I understand how you computed but for older nbms I don't believe we
>> will have even a fraction of those maximum downloads.
>>
>> --emi
>>
>> http://coolbeans.xyz/ - CoolBeans: An IDE for Java, JavaEE, PHP and more!
>>
>> On Thu, Dec 27, 2018 at 10:44 AM Laszlo Kishalmi
>> <la...@gmail.com> wrote:
>>>
>>> On 12/26/18 11:23 PM, Emilian Bold wrote:
>>>> Considering the are no updates, why would we even serve the nbms? To
>>>> older 8.2 installs or what? Won't those older installs request the 10
>>>> nbms anyhow? So, I don't see how those nbms would get so many
>>>> downloads.
>>>
>>> I've just really calculated the worst case scenario for the download
>>> traffic:
>>>
>>> 1. Number of request per month on updates.xml.gz is 22 - 25 thousand I
>>> took 25000
>>> 2. NetBeans 10.0 nbm-s take more space (118Mb) than 9.0, so I
>>> calculated with 120Mb
>>>
>>> So if everyone would download the whole nbm structure whenever it hits
>>> the updates.xml.gz, then we could have 25000 * 120 Mb = 3 Tb per month
>>> download traffic.
>>>
>>>> We can put the smaller updates.xml.gz anywhere, it won't make any
>>>> impact on bandwidth.
>>>>
>>>> I would not use any 3rd party solution unless Apache infra tell us so.
>>>> The one thing Apache provides to projects is infrastructure.
>>> Well, Apache provides more than infra, however that infra not necessary
>>> designed for a project like us., that are producing 650+ artifacts,
>>> weighs ~500 Mb with every release.
>>>>
>>>> I say we pick the simplest solution: use archive.apache.org for
>>>> everything. It will serve only the updates.xml.gz in reality and if
>>>> they do get too much traffic infra will let us know and we figure
>>>> something out then.
>>> Well, that's might be the easiest solution, still option 2 or 3 would be
>>> easy to implement as well and clean up our project layout on
>>> dist.apache.org.
>>>>
>>>> BTW, I still believe we have to create, sign and store the nbms, even
>>>> if they are only convenience binaries.
>>> That's no questions. My only concern is where to store.
>>>>
>>>> --emi
>>>>
>>>> http://coolbeans.xyz/ - CoolBeans: An IDE for Java, JavaEE, PHP and
>>>> more!
>>>>
>>>> On Thu, Dec 27, 2018 at 8:11 AM Laszlo Kishalmi
>>>> <la...@gmail.com> wrote:
>>>>> Dear all,
>>>>>
>>>>> I have to bring up this old item again. According to the Apache
>>>>> release
>>>>> processes, I'd need to archive the current 9.0 release from the site.
>>>>> this means that our 9.0 NBM-s are going to be removed from there as
>>>>> well. The files are going to be available on archive.apache.org
>>>>> which is
>>>>> not mirrored. Only the most recent release are supposed to be out
>>>>> on the
>>>>> release and their mirrors.
>>>>>
>>>>> This means we need to find some place to our 9.0 nbms. I also won't
>>>>> mind
>>>>> if we could move away storing our nbms, "officially" (there are no
>>>>> official binary releases, yet) on some other place than Apache dist
>>>>> infrastructure. It's a bit weird to sign and upload 650+ binaries with
>>>>> each and every release. So let's go back to square one.
>>>>>
>>>>> 1. We can change the redirection to the Apache archive site.
>>>>> That is
>>>>> not mirrored, and initially we would put a real traffic on
>>>>> there.
>>>>> 2. We can serve the nbm-s from our netbeans-vm.
>>>>> Checked the stats we get ~25000 requests a month on
>>>>> updates.xml.gz
>>>>> if we'd transfer all the update center (~120 Mb, for NetBeans
>>>>> 10.0)
>>>>> all the modules with each request for updates.xml.gz our
>>>>> transfer
>>>>> would be ~3 Tb. So we would definitely fit into the 3-5
>>>>> Tb/per month
>>>>> usage limit. That would be actually much less, as this is a
>>>>> worst
>>>>> case scenario.
>>>>> 3. Search for a free third party solution.
>>>>> I'd try Bintray, we can get 10 Gb space and CDN, so it is pretty
>>>>> reliable, the only concern is the 1 Tb of network limit per
>>>>> month.
>>>>> Unfortunately I only can guess that the 3 Tb is really
>>>>> exaggerated.
>>>>> All the log files I've found do not include the actual download
>>>>> statistics of our nbm-s
>>>>> 4. Something else?
>>>>>
>>>>> I can't archive the current 9.0 till this thing is decided.
>>>>>
>>>>> On 4/10/18 11:39 PM, Antonio wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> In order to understand this long thread (about NETBEANS-330 [1]) I
>>>>>> tried to summarize it. Please review and send corrections & questions
>>>>>> as appropriate.
>>>>>>
>>>>>> Kind regards,
>>>>>> Antonio
>>>>>>
>>>>>> == Objectives
>>>>>>
>>>>>> Host the NetBeans 9.0 Update Center on Apache infrastructure.
>>>>>>
>>>>>> == Constraints/facts/options
>>>>>>
>>>>>> - We cannot host it on a website due to bandwidth requirements of 3-5
>>>>>> Tb/month.
>>>>>>
>>>>>> - We must use the Apache Mirror system and their "closer.cgi,
>>>>>> closer.lua" cgi scripts [2] to select the closest mirror to the user.
>>>>>> This script can either redirect directly to the closest binary file
>>>>>> (ready to download), or return a JSON response with a list of closest
>>>>>> mirrors.
>>>>>>
>>>>>> - Distributing through the Apache Mirror system requires a proper
>>>>>> release, with voting, approval and signing, etc.
>>>>>>
>>>>>> - Geertjan says that prior to the final release we could do an rc
>>>>>> release to have this feature tested.
>>>>>>
>>>>>> - The UC catalog xml file can be hosted in the mirror system. For
>>>>>> this
>>>>>> to work the catalog xml file must contain relative paths, so when
>>>>>> aged
>>>>>> releases are moved away from the mirror system and into the archive
>>>>>> things keep working.
>>>>>>
>>>>>> - We can HTTP redirect to the proper catalog.xml file...
>>>>>>
>>>>>> a) ... From our website, redirecting to the Apache mirror system
>>>>>> or to
>>>>>> the archive with an .htaccess file (under our control).
>>>>>>
>>>>>> b) ... Idem, by asking infra to modify our server configuration (more
>>>>>> performant but requires Infra tickets).
>>>>>>
>>>>>> c) ... Using a script of ours hosted at the "VM" (a web server ours
>>>>>> currently hosting selenium), that may also track/log some statistics.
>>>>>>
>>>>>> - The NetBeans UC module will make a request for the XML file to our
>>>>>> website, from there will be redirected to the mirror/archive.
>>>>>> Later on
>>>>>> it will build up proper URLs (using some prefix and the relative
>>>>>> paths
>>>>>> inside the catalog xml file) for the final NBM downloads.
>>>>>>
>>>>>> - Mirror download statistics may be currently downloaded from
>>>>>> https://www.apache.org/dyn/stats/netbeans.log
>>>>>>
>>>>>> - Web server statistics are available through a very detailed request
>>>>>> to infra. They say they're not in the "counting business" :-)
>>>>>>
>>>>>> - Jan is working in NBM generation (with NOTICE & LICENSE, etc.).
>>>>>> Jenkins now generates these artifacts:
>>>>>> https://builds.apache.org/view/Incubator%20Projects/job/incubator-netbeans-linux/lastSuccessfulBuild/artifact/nbbuild/nbms/
>>>>>>
>>>>>>
>>>>>>
>>>>>> == Open issues & questions
>>>>>>
>>>>>> 1- It seems we won't be able to host some OSGi bundles in the mirror
>>>>>> system due to licensing issues?
>>>>>>
>>>>>> 2- Mirrors are not as reliable as a CDN: files can be corrupt, or
>>>>>> whatever. The UC module should verify the integrity of the downloaded
>>>>>> files, etc.
>>>>>>
>>>>>> 3- We may want to select a small set of NBM files for the first run,
>>>>>> to workaround previous licensing problems.
>>>>>>
>>>>>>
>>>>>> [1]
>>>>>> https://issues.apache.org/jira/browse/NETBEANS-330
>>>>>>
>>>>>> [2]
>>>>>> https://reference.apache.org/pmc/mirror_scripts
>>>>>>
>>>>>>
>>>>>> On 05/04/18 14:38, Geertjan Wielenga wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> We need to nail down this one and I think the key blocker is that
>>>>>>> there are
>>>>>>> different ideas about what this is about:
>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/NETBEANS-330
>>>>>>>
>>>>>>> The above is not about the Plugin Portal.
>>>>>>>
>>>>>>> If I understand it correctly, this is about where the NBMs (which
>>>>>>> ones? how
>>>>>>> many? do we know?) and the related XML file (a.k.a. update center)
>>>>>>> will be
>>>>>>> hosted.
>>>>>>>
>>>>>>> AFAIK, the XML file and the NBMs could be put onto our Apache
>>>>>>> NetBeans VM
>>>>>>> just like Synergy:
>>>>>>>
>>>>>>> http://netbeans-vm.apache.org/synergy
>>>>>>>
>>>>>>> The key question remains, which NBMs are we talking about here,
>>>>>>> applicable
>>>>>>> to the 9.0 release, I think.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Gj
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
>>>>>> For additional commands, e-mail:
>>>>>> dev-help@netbeans.incubator.apache.org
>>>>>>
>>>>>> For further information about the NetBeans mailing lists, visit:
>>>>>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>>>>>>
>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
>>>> For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
>>>>
>>>> For further information about the NetBeans mailing lists, visit:
>>>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>>>>
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
>> For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
>>
>> For further information about the NetBeans mailing lists, visit:
>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>>
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Resurrect: Summary of How/where exactly to host the NBMs for the
9.0 release
Posted by Antonio <an...@vieiro.net>.
Hi all,
I'm asking at infra.chat if we should use the archive or
netbeans-vm.apache.org.
Cheers,
Antonio
El 27/12/2018 a las 9:46, Emilian Bold escribió:
>> That's no questions. My only concern is where to store.
>
> We store on Apache infra and if it's too much for them we ask them for
> suggestions or new servers.
>
> I understand how you computed but for older nbms I don't believe we
> will have even a fraction of those maximum downloads.
>
> --emi
>
> http://coolbeans.xyz/ - CoolBeans: An IDE for Java, JavaEE, PHP and more!
>
> On Thu, Dec 27, 2018 at 10:44 AM Laszlo Kishalmi
> <la...@gmail.com> wrote:
>>
>> On 12/26/18 11:23 PM, Emilian Bold wrote:
>>> Considering the are no updates, why would we even serve the nbms? To
>>> older 8.2 installs or what? Won't those older installs request the 10
>>> nbms anyhow? So, I don't see how those nbms would get so many
>>> downloads.
>>
>> I've just really calculated the worst case scenario for the download
>> traffic:
>>
>> 1. Number of request per month on updates.xml.gz is 22 - 25 thousand I
>> took 25000
>> 2. NetBeans 10.0 nbm-s take more space (118Mb) than 9.0, so I
>> calculated with 120Mb
>>
>> So if everyone would download the whole nbm structure whenever it hits
>> the updates.xml.gz, then we could have 25000 * 120 Mb = 3 Tb per month
>> download traffic.
>>
>>> We can put the smaller updates.xml.gz anywhere, it won't make any
>>> impact on bandwidth.
>>>
>>> I would not use any 3rd party solution unless Apache infra tell us so.
>>> The one thing Apache provides to projects is infrastructure.
>> Well, Apache provides more than infra, however that infra not necessary
>> designed for a project like us., that are producing 650+ artifacts,
>> weighs ~500 Mb with every release.
>>>
>>> I say we pick the simplest solution: use archive.apache.org for
>>> everything. It will serve only the updates.xml.gz in reality and if
>>> they do get too much traffic infra will let us know and we figure
>>> something out then.
>> Well, that's might be the easiest solution, still option 2 or 3 would be
>> easy to implement as well and clean up our project layout on
>> dist.apache.org.
>>>
>>> BTW, I still believe we have to create, sign and store the nbms, even
>>> if they are only convenience binaries.
>> That's no questions. My only concern is where to store.
>>>
>>> --emi
>>>
>>> http://coolbeans.xyz/ - CoolBeans: An IDE for Java, JavaEE, PHP and more!
>>>
>>> On Thu, Dec 27, 2018 at 8:11 AM Laszlo Kishalmi
>>> <la...@gmail.com> wrote:
>>>> Dear all,
>>>>
>>>> I have to bring up this old item again. According to the Apache release
>>>> processes, I'd need to archive the current 9.0 release from the site.
>>>> this means that our 9.0 NBM-s are going to be removed from there as
>>>> well. The files are going to be available on archive.apache.org which is
>>>> not mirrored. Only the most recent release are supposed to be out on the
>>>> release and their mirrors.
>>>>
>>>> This means we need to find some place to our 9.0 nbms. I also won't mind
>>>> if we could move away storing our nbms, "officially" (there are no
>>>> official binary releases, yet) on some other place than Apache dist
>>>> infrastructure. It's a bit weird to sign and upload 650+ binaries with
>>>> each and every release. So let's go back to square one.
>>>>
>>>> 1. We can change the redirection to the Apache archive site. That is
>>>> not mirrored, and initially we would put a real traffic on there.
>>>> 2. We can serve the nbm-s from our netbeans-vm.
>>>> Checked the stats we get ~25000 requests a month on updates.xml.gz
>>>> if we'd transfer all the update center (~120 Mb, for NetBeans 10.0)
>>>> all the modules with each request for updates.xml.gz our transfer
>>>> would be ~3 Tb. So we would definitely fit into the 3-5 Tb/per month
>>>> usage limit. That would be actually much less, as this is a worst
>>>> case scenario.
>>>> 3. Search for a free third party solution.
>>>> I'd try Bintray, we can get 10 Gb space and CDN, so it is pretty
>>>> reliable, the only concern is the 1 Tb of network limit per month.
>>>> Unfortunately I only can guess that the 3 Tb is really exaggerated.
>>>> All the log files I've found do not include the actual download
>>>> statistics of our nbm-s
>>>> 4. Something else?
>>>>
>>>> I can't archive the current 9.0 till this thing is decided.
>>>>
>>>> On 4/10/18 11:39 PM, Antonio wrote:
>>>>> Hi all,
>>>>>
>>>>> In order to understand this long thread (about NETBEANS-330 [1]) I
>>>>> tried to summarize it. Please review and send corrections & questions
>>>>> as appropriate.
>>>>>
>>>>> Kind regards,
>>>>> Antonio
>>>>>
>>>>> == Objectives
>>>>>
>>>>> Host the NetBeans 9.0 Update Center on Apache infrastructure.
>>>>>
>>>>> == Constraints/facts/options
>>>>>
>>>>> - We cannot host it on a website due to bandwidth requirements of 3-5
>>>>> Tb/month.
>>>>>
>>>>> - We must use the Apache Mirror system and their "closer.cgi,
>>>>> closer.lua" cgi scripts [2] to select the closest mirror to the user.
>>>>> This script can either redirect directly to the closest binary file
>>>>> (ready to download), or return a JSON response with a list of closest
>>>>> mirrors.
>>>>>
>>>>> - Distributing through the Apache Mirror system requires a proper
>>>>> release, with voting, approval and signing, etc.
>>>>>
>>>>> - Geertjan says that prior to the final release we could do an rc
>>>>> release to have this feature tested.
>>>>>
>>>>> - The UC catalog xml file can be hosted in the mirror system. For this
>>>>> to work the catalog xml file must contain relative paths, so when aged
>>>>> releases are moved away from the mirror system and into the archive
>>>>> things keep working.
>>>>>
>>>>> - We can HTTP redirect to the proper catalog.xml file...
>>>>>
>>>>> a) ... From our website, redirecting to the Apache mirror system or to
>>>>> the archive with an .htaccess file (under our control).
>>>>>
>>>>> b) ... Idem, by asking infra to modify our server configuration (more
>>>>> performant but requires Infra tickets).
>>>>>
>>>>> c) ... Using a script of ours hosted at the "VM" (a web server ours
>>>>> currently hosting selenium), that may also track/log some statistics.
>>>>>
>>>>> - The NetBeans UC module will make a request for the XML file to our
>>>>> website, from there will be redirected to the mirror/archive. Later on
>>>>> it will build up proper URLs (using some prefix and the relative paths
>>>>> inside the catalog xml file) for the final NBM downloads.
>>>>>
>>>>> - Mirror download statistics may be currently downloaded from
>>>>> https://www.apache.org/dyn/stats/netbeans.log
>>>>>
>>>>> - Web server statistics are available through a very detailed request
>>>>> to infra. They say they're not in the "counting business" :-)
>>>>>
>>>>> - Jan is working in NBM generation (with NOTICE & LICENSE, etc.).
>>>>> Jenkins now generates these artifacts:
>>>>> https://builds.apache.org/view/Incubator%20Projects/job/incubator-netbeans-linux/lastSuccessfulBuild/artifact/nbbuild/nbms/
>>>>>
>>>>>
>>>>> == Open issues & questions
>>>>>
>>>>> 1- It seems we won't be able to host some OSGi bundles in the mirror
>>>>> system due to licensing issues?
>>>>>
>>>>> 2- Mirrors are not as reliable as a CDN: files can be corrupt, or
>>>>> whatever. The UC module should verify the integrity of the downloaded
>>>>> files, etc.
>>>>>
>>>>> 3- We may want to select a small set of NBM files for the first run,
>>>>> to workaround previous licensing problems.
>>>>>
>>>>>
>>>>> [1]
>>>>> https://issues.apache.org/jira/browse/NETBEANS-330
>>>>>
>>>>> [2]
>>>>> https://reference.apache.org/pmc/mirror_scripts
>>>>>
>>>>>
>>>>> On 05/04/18 14:38, Geertjan Wielenga wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> We need to nail down this one and I think the key blocker is that
>>>>>> there are
>>>>>> different ideas about what this is about:
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/NETBEANS-330
>>>>>>
>>>>>> The above is not about the Plugin Portal.
>>>>>>
>>>>>> If I understand it correctly, this is about where the NBMs (which
>>>>>> ones? how
>>>>>> many? do we know?) and the related XML file (a.k.a. update center)
>>>>>> will be
>>>>>> hosted.
>>>>>>
>>>>>> AFAIK, the XML file and the NBMs could be put onto our Apache
>>>>>> NetBeans VM
>>>>>> just like Synergy:
>>>>>>
>>>>>> http://netbeans-vm.apache.org/synergy
>>>>>>
>>>>>> The key question remains, which NBMs are we talking about here,
>>>>>> applicable
>>>>>> to the 9.0 release, I think.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Gj
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
>>>>> For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
>>>>>
>>>>> For further information about the NetBeans mailing lists, visit:
>>>>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>>>>>
>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
>>> For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
>>>
>>> For further information about the NetBeans mailing lists, visit:
>>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
> For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Resurrect: Summary of How/where exactly to host the NBMs for the
9.0 release
Posted by Emilian Bold <em...@gmail.com>.
> That's no questions. My only concern is where to store.
We store on Apache infra and if it's too much for them we ask them for
suggestions or new servers.
I understand how you computed but for older nbms I don't believe we
will have even a fraction of those maximum downloads.
--emi
http://coolbeans.xyz/ - CoolBeans: An IDE for Java, JavaEE, PHP and more!
On Thu, Dec 27, 2018 at 10:44 AM Laszlo Kishalmi
<la...@gmail.com> wrote:
>
> On 12/26/18 11:23 PM, Emilian Bold wrote:
> > Considering the are no updates, why would we even serve the nbms? To
> > older 8.2 installs or what? Won't those older installs request the 10
> > nbms anyhow? So, I don't see how those nbms would get so many
> > downloads.
>
> I've just really calculated the worst case scenario for the download
> traffic:
>
> 1. Number of request per month on updates.xml.gz is 22 - 25 thousand I
> took 25000
> 2. NetBeans 10.0 nbm-s take more space (118Mb) than 9.0, so I
> calculated with 120Mb
>
> So if everyone would download the whole nbm structure whenever it hits
> the updates.xml.gz, then we could have 25000 * 120 Mb = 3 Tb per month
> download traffic.
>
> > We can put the smaller updates.xml.gz anywhere, it won't make any
> > impact on bandwidth.
> >
> > I would not use any 3rd party solution unless Apache infra tell us so.
> > The one thing Apache provides to projects is infrastructure.
> Well, Apache provides more than infra, however that infra not necessary
> designed for a project like us., that are producing 650+ artifacts,
> weighs ~500 Mb with every release.
> >
> > I say we pick the simplest solution: use archive.apache.org for
> > everything. It will serve only the updates.xml.gz in reality and if
> > they do get too much traffic infra will let us know and we figure
> > something out then.
> Well, that's might be the easiest solution, still option 2 or 3 would be
> easy to implement as well and clean up our project layout on
> dist.apache.org.
> >
> > BTW, I still believe we have to create, sign and store the nbms, even
> > if they are only convenience binaries.
> That's no questions. My only concern is where to store.
> >
> > --emi
> >
> > http://coolbeans.xyz/ - CoolBeans: An IDE for Java, JavaEE, PHP and more!
> >
> > On Thu, Dec 27, 2018 at 8:11 AM Laszlo Kishalmi
> > <la...@gmail.com> wrote:
> >> Dear all,
> >>
> >> I have to bring up this old item again. According to the Apache release
> >> processes, I'd need to archive the current 9.0 release from the site.
> >> this means that our 9.0 NBM-s are going to be removed from there as
> >> well. The files are going to be available on archive.apache.org which is
> >> not mirrored. Only the most recent release are supposed to be out on the
> >> release and their mirrors.
> >>
> >> This means we need to find some place to our 9.0 nbms. I also won't mind
> >> if we could move away storing our nbms, "officially" (there are no
> >> official binary releases, yet) on some other place than Apache dist
> >> infrastructure. It's a bit weird to sign and upload 650+ binaries with
> >> each and every release. So let's go back to square one.
> >>
> >> 1. We can change the redirection to the Apache archive site. That is
> >> not mirrored, and initially we would put a real traffic on there.
> >> 2. We can serve the nbm-s from our netbeans-vm.
> >> Checked the stats we get ~25000 requests a month on updates.xml.gz
> >> if we'd transfer all the update center (~120 Mb, for NetBeans 10.0)
> >> all the modules with each request for updates.xml.gz our transfer
> >> would be ~3 Tb. So we would definitely fit into the 3-5 Tb/per month
> >> usage limit. That would be actually much less, as this is a worst
> >> case scenario.
> >> 3. Search for a free third party solution.
> >> I'd try Bintray, we can get 10 Gb space and CDN, so it is pretty
> >> reliable, the only concern is the 1 Tb of network limit per month.
> >> Unfortunately I only can guess that the 3 Tb is really exaggerated.
> >> All the log files I've found do not include the actual download
> >> statistics of our nbm-s
> >> 4. Something else?
> >>
> >> I can't archive the current 9.0 till this thing is decided.
> >>
> >> On 4/10/18 11:39 PM, Antonio wrote:
> >>> Hi all,
> >>>
> >>> In order to understand this long thread (about NETBEANS-330 [1]) I
> >>> tried to summarize it. Please review and send corrections & questions
> >>> as appropriate.
> >>>
> >>> Kind regards,
> >>> Antonio
> >>>
> >>> == Objectives
> >>>
> >>> Host the NetBeans 9.0 Update Center on Apache infrastructure.
> >>>
> >>> == Constraints/facts/options
> >>>
> >>> - We cannot host it on a website due to bandwidth requirements of 3-5
> >>> Tb/month.
> >>>
> >>> - We must use the Apache Mirror system and their "closer.cgi,
> >>> closer.lua" cgi scripts [2] to select the closest mirror to the user.
> >>> This script can either redirect directly to the closest binary file
> >>> (ready to download), or return a JSON response with a list of closest
> >>> mirrors.
> >>>
> >>> - Distributing through the Apache Mirror system requires a proper
> >>> release, with voting, approval and signing, etc.
> >>>
> >>> - Geertjan says that prior to the final release we could do an rc
> >>> release to have this feature tested.
> >>>
> >>> - The UC catalog xml file can be hosted in the mirror system. For this
> >>> to work the catalog xml file must contain relative paths, so when aged
> >>> releases are moved away from the mirror system and into the archive
> >>> things keep working.
> >>>
> >>> - We can HTTP redirect to the proper catalog.xml file...
> >>>
> >>> a) ... From our website, redirecting to the Apache mirror system or to
> >>> the archive with an .htaccess file (under our control).
> >>>
> >>> b) ... Idem, by asking infra to modify our server configuration (more
> >>> performant but requires Infra tickets).
> >>>
> >>> c) ... Using a script of ours hosted at the "VM" (a web server ours
> >>> currently hosting selenium), that may also track/log some statistics.
> >>>
> >>> - The NetBeans UC module will make a request for the XML file to our
> >>> website, from there will be redirected to the mirror/archive. Later on
> >>> it will build up proper URLs (using some prefix and the relative paths
> >>> inside the catalog xml file) for the final NBM downloads.
> >>>
> >>> - Mirror download statistics may be currently downloaded from
> >>> https://www.apache.org/dyn/stats/netbeans.log
> >>>
> >>> - Web server statistics are available through a very detailed request
> >>> to infra. They say they're not in the "counting business" :-)
> >>>
> >>> - Jan is working in NBM generation (with NOTICE & LICENSE, etc.).
> >>> Jenkins now generates these artifacts:
> >>> https://builds.apache.org/view/Incubator%20Projects/job/incubator-netbeans-linux/lastSuccessfulBuild/artifact/nbbuild/nbms/
> >>>
> >>>
> >>> == Open issues & questions
> >>>
> >>> 1- It seems we won't be able to host some OSGi bundles in the mirror
> >>> system due to licensing issues?
> >>>
> >>> 2- Mirrors are not as reliable as a CDN: files can be corrupt, or
> >>> whatever. The UC module should verify the integrity of the downloaded
> >>> files, etc.
> >>>
> >>> 3- We may want to select a small set of NBM files for the first run,
> >>> to workaround previous licensing problems.
> >>>
> >>>
> >>> [1]
> >>> https://issues.apache.org/jira/browse/NETBEANS-330
> >>>
> >>> [2]
> >>> https://reference.apache.org/pmc/mirror_scripts
> >>>
> >>>
> >>> On 05/04/18 14:38, Geertjan Wielenga wrote:
> >>>> Hi all,
> >>>>
> >>>> We need to nail down this one and I think the key blocker is that
> >>>> there are
> >>>> different ideas about what this is about:
> >>>>
> >>>> https://issues.apache.org/jira/browse/NETBEANS-330
> >>>>
> >>>> The above is not about the Plugin Portal.
> >>>>
> >>>> If I understand it correctly, this is about where the NBMs (which
> >>>> ones? how
> >>>> many? do we know?) and the related XML file (a.k.a. update center)
> >>>> will be
> >>>> hosted.
> >>>>
> >>>> AFAIK, the XML file and the NBMs could be put onto our Apache
> >>>> NetBeans VM
> >>>> just like Synergy:
> >>>>
> >>>> http://netbeans-vm.apache.org/synergy
> >>>>
> >>>> The key question remains, which NBMs are we talking about here,
> >>>> applicable
> >>>> to the 9.0 release, I think.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Gj
> >>>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
> >>> For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
> >>>
> >>> For further information about the NetBeans mailing lists, visit:
> >>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
> >>>
> >>>
> >>>
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
> > For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
> >
> > For further information about the NetBeans mailing lists, visit:
> > https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
> >
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Resurrect: Summary of How/where exactly to host the NBMs for the
9.0 release
Posted by Laszlo Kishalmi <la...@gmail.com>.
On 12/26/18 11:23 PM, Emilian Bold wrote:
> Considering the are no updates, why would we even serve the nbms? To
> older 8.2 installs or what? Won't those older installs request the 10
> nbms anyhow? So, I don't see how those nbms would get so many
> downloads.
I've just really calculated the worst case scenario for the download
traffic:
1. Number of request per month on updates.xml.gz is 22 - 25 thousand I
took 25000
2. NetBeans 10.0 nbm-s take more space (118Mb) than 9.0, so I
calculated with 120Mb
So if everyone would download the whole nbm structure whenever it hits
the updates.xml.gz, then we could have 25000 * 120 Mb = 3 Tb per month
download traffic.
> We can put the smaller updates.xml.gz anywhere, it won't make any
> impact on bandwidth.
>
> I would not use any 3rd party solution unless Apache infra tell us so.
> The one thing Apache provides to projects is infrastructure.
Well, Apache provides more than infra, however that infra not necessary
designed for a project like us., that are producing 650+ artifacts,
weighs ~500 Mb with every release.
>
> I say we pick the simplest solution: use archive.apache.org for
> everything. It will serve only the updates.xml.gz in reality and if
> they do get too much traffic infra will let us know and we figure
> something out then.
Well, that's might be the easiest solution, still option 2 or 3 would be
easy to implement as well and clean up our project layout on
dist.apache.org.
>
> BTW, I still believe we have to create, sign and store the nbms, even
> if they are only convenience binaries.
That's no questions. My only concern is where to store.
>
> --emi
>
> http://coolbeans.xyz/ - CoolBeans: An IDE for Java, JavaEE, PHP and more!
>
> On Thu, Dec 27, 2018 at 8:11 AM Laszlo Kishalmi
> <la...@gmail.com> wrote:
>> Dear all,
>>
>> I have to bring up this old item again. According to the Apache release
>> processes, I'd need to archive the current 9.0 release from the site.
>> this means that our 9.0 NBM-s are going to be removed from there as
>> well. The files are going to be available on archive.apache.org which is
>> not mirrored. Only the most recent release are supposed to be out on the
>> release and their mirrors.
>>
>> This means we need to find some place to our 9.0 nbms. I also won't mind
>> if we could move away storing our nbms, "officially" (there are no
>> official binary releases, yet) on some other place than Apache dist
>> infrastructure. It's a bit weird to sign and upload 650+ binaries with
>> each and every release. So let's go back to square one.
>>
>> 1. We can change the redirection to the Apache archive site. That is
>> not mirrored, and initially we would put a real traffic on there.
>> 2. We can serve the nbm-s from our netbeans-vm.
>> Checked the stats we get ~25000 requests a month on updates.xml.gz
>> if we'd transfer all the update center (~120 Mb, for NetBeans 10.0)
>> all the modules with each request for updates.xml.gz our transfer
>> would be ~3 Tb. So we would definitely fit into the 3-5 Tb/per month
>> usage limit. That would be actually much less, as this is a worst
>> case scenario.
>> 3. Search for a free third party solution.
>> I'd try Bintray, we can get 10 Gb space and CDN, so it is pretty
>> reliable, the only concern is the 1 Tb of network limit per month.
>> Unfortunately I only can guess that the 3 Tb is really exaggerated.
>> All the log files I've found do not include the actual download
>> statistics of our nbm-s
>> 4. Something else?
>>
>> I can't archive the current 9.0 till this thing is decided.
>>
>> On 4/10/18 11:39 PM, Antonio wrote:
>>> Hi all,
>>>
>>> In order to understand this long thread (about NETBEANS-330 [1]) I
>>> tried to summarize it. Please review and send corrections & questions
>>> as appropriate.
>>>
>>> Kind regards,
>>> Antonio
>>>
>>> == Objectives
>>>
>>> Host the NetBeans 9.0 Update Center on Apache infrastructure.
>>>
>>> == Constraints/facts/options
>>>
>>> - We cannot host it on a website due to bandwidth requirements of 3-5
>>> Tb/month.
>>>
>>> - We must use the Apache Mirror system and their "closer.cgi,
>>> closer.lua" cgi scripts [2] to select the closest mirror to the user.
>>> This script can either redirect directly to the closest binary file
>>> (ready to download), or return a JSON response with a list of closest
>>> mirrors.
>>>
>>> - Distributing through the Apache Mirror system requires a proper
>>> release, with voting, approval and signing, etc.
>>>
>>> - Geertjan says that prior to the final release we could do an rc
>>> release to have this feature tested.
>>>
>>> - The UC catalog xml file can be hosted in the mirror system. For this
>>> to work the catalog xml file must contain relative paths, so when aged
>>> releases are moved away from the mirror system and into the archive
>>> things keep working.
>>>
>>> - We can HTTP redirect to the proper catalog.xml file...
>>>
>>> a) ... From our website, redirecting to the Apache mirror system or to
>>> the archive with an .htaccess file (under our control).
>>>
>>> b) ... Idem, by asking infra to modify our server configuration (more
>>> performant but requires Infra tickets).
>>>
>>> c) ... Using a script of ours hosted at the "VM" (a web server ours
>>> currently hosting selenium), that may also track/log some statistics.
>>>
>>> - The NetBeans UC module will make a request for the XML file to our
>>> website, from there will be redirected to the mirror/archive. Later on
>>> it will build up proper URLs (using some prefix and the relative paths
>>> inside the catalog xml file) for the final NBM downloads.
>>>
>>> - Mirror download statistics may be currently downloaded from
>>> https://www.apache.org/dyn/stats/netbeans.log
>>>
>>> - Web server statistics are available through a very detailed request
>>> to infra. They say they're not in the "counting business" :-)
>>>
>>> - Jan is working in NBM generation (with NOTICE & LICENSE, etc.).
>>> Jenkins now generates these artifacts:
>>> https://builds.apache.org/view/Incubator%20Projects/job/incubator-netbeans-linux/lastSuccessfulBuild/artifact/nbbuild/nbms/
>>>
>>>
>>> == Open issues & questions
>>>
>>> 1- It seems we won't be able to host some OSGi bundles in the mirror
>>> system due to licensing issues?
>>>
>>> 2- Mirrors are not as reliable as a CDN: files can be corrupt, or
>>> whatever. The UC module should verify the integrity of the downloaded
>>> files, etc.
>>>
>>> 3- We may want to select a small set of NBM files for the first run,
>>> to workaround previous licensing problems.
>>>
>>>
>>> [1]
>>> https://issues.apache.org/jira/browse/NETBEANS-330
>>>
>>> [2]
>>> https://reference.apache.org/pmc/mirror_scripts
>>>
>>>
>>> On 05/04/18 14:38, Geertjan Wielenga wrote:
>>>> Hi all,
>>>>
>>>> We need to nail down this one and I think the key blocker is that
>>>> there are
>>>> different ideas about what this is about:
>>>>
>>>> https://issues.apache.org/jira/browse/NETBEANS-330
>>>>
>>>> The above is not about the Plugin Portal.
>>>>
>>>> If I understand it correctly, this is about where the NBMs (which
>>>> ones? how
>>>> many? do we know?) and the related XML file (a.k.a. update center)
>>>> will be
>>>> hosted.
>>>>
>>>> AFAIK, the XML file and the NBMs could be put onto our Apache
>>>> NetBeans VM
>>>> just like Synergy:
>>>>
>>>> http://netbeans-vm.apache.org/synergy
>>>>
>>>> The key question remains, which NBMs are we talking about here,
>>>> applicable
>>>> to the 9.0 release, I think.
>>>>
>>>> Thanks,
>>>>
>>>> Gj
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
>>> For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
>>>
>>> For further information about the NetBeans mailing lists, visit:
>>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>>>
>>>
>>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
> For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
Re: Resurrect: Summary of How/where exactly to host the NBMs for the
9.0 release
Posted by Emilian Bold <em...@gmail.com>.
Considering the are no updates, why would we even serve the nbms? To
older 8.2 installs or what? Won't those older installs request the 10
nbms anyhow? So, I don't see how those nbms would get so many
downloads.
We can put the smaller updates.xml.gz anywhere, it won't make any
impact on bandwidth.
I would not use any 3rd party solution unless Apache infra tell us so.
The one thing Apache provides to projects is infrastructure.
I say we pick the simplest solution: use archive.apache.org for
everything. It will serve only the updates.xml.gz in reality and if
they do get too much traffic infra will let us know and we figure
something out then.
BTW, I still believe we have to create, sign and store the nbms, even
if they are only convenience binaries.
--emi
http://coolbeans.xyz/ - CoolBeans: An IDE for Java, JavaEE, PHP and more!
On Thu, Dec 27, 2018 at 8:11 AM Laszlo Kishalmi
<la...@gmail.com> wrote:
>
> Dear all,
>
> I have to bring up this old item again. According to the Apache release
> processes, I'd need to archive the current 9.0 release from the site.
> this means that our 9.0 NBM-s are going to be removed from there as
> well. The files are going to be available on archive.apache.org which is
> not mirrored. Only the most recent release are supposed to be out on the
> release and their mirrors.
>
> This means we need to find some place to our 9.0 nbms. I also won't mind
> if we could move away storing our nbms, "officially" (there are no
> official binary releases, yet) on some other place than Apache dist
> infrastructure. It's a bit weird to sign and upload 650+ binaries with
> each and every release. So let's go back to square one.
>
> 1. We can change the redirection to the Apache archive site. That is
> not mirrored, and initially we would put a real traffic on there.
> 2. We can serve the nbm-s from our netbeans-vm.
> Checked the stats we get ~25000 requests a month on updates.xml.gz
> if we'd transfer all the update center (~120 Mb, for NetBeans 10.0)
> all the modules with each request for updates.xml.gz our transfer
> would be ~3 Tb. So we would definitely fit into the 3-5 Tb/per month
> usage limit. That would be actually much less, as this is a worst
> case scenario.
> 3. Search for a free third party solution.
> I'd try Bintray, we can get 10 Gb space and CDN, so it is pretty
> reliable, the only concern is the 1 Tb of network limit per month.
> Unfortunately I only can guess that the 3 Tb is really exaggerated.
> All the log files I've found do not include the actual download
> statistics of our nbm-s
> 4. Something else?
>
> I can't archive the current 9.0 till this thing is decided.
>
> On 4/10/18 11:39 PM, Antonio wrote:
> > Hi all,
> >
> > In order to understand this long thread (about NETBEANS-330 [1]) I
> > tried to summarize it. Please review and send corrections & questions
> > as appropriate.
> >
> > Kind regards,
> > Antonio
> >
> > == Objectives
> >
> > Host the NetBeans 9.0 Update Center on Apache infrastructure.
> >
> > == Constraints/facts/options
> >
> > - We cannot host it on a website due to bandwidth requirements of 3-5
> > Tb/month.
> >
> > - We must use the Apache Mirror system and their "closer.cgi,
> > closer.lua" cgi scripts [2] to select the closest mirror to the user.
> > This script can either redirect directly to the closest binary file
> > (ready to download), or return a JSON response with a list of closest
> > mirrors.
> >
> > - Distributing through the Apache Mirror system requires a proper
> > release, with voting, approval and signing, etc.
> >
> > - Geertjan says that prior to the final release we could do an rc
> > release to have this feature tested.
> >
> > - The UC catalog xml file can be hosted in the mirror system. For this
> > to work the catalog xml file must contain relative paths, so when aged
> > releases are moved away from the mirror system and into the archive
> > things keep working.
> >
> > - We can HTTP redirect to the proper catalog.xml file...
> >
> > a) ... From our website, redirecting to the Apache mirror system or to
> > the archive with an .htaccess file (under our control).
> >
> > b) ... Idem, by asking infra to modify our server configuration (more
> > performant but requires Infra tickets).
> >
> > c) ... Using a script of ours hosted at the "VM" (a web server ours
> > currently hosting selenium), that may also track/log some statistics.
> >
> > - The NetBeans UC module will make a request for the XML file to our
> > website, from there will be redirected to the mirror/archive. Later on
> > it will build up proper URLs (using some prefix and the relative paths
> > inside the catalog xml file) for the final NBM downloads.
> >
> > - Mirror download statistics may be currently downloaded from
> > https://www.apache.org/dyn/stats/netbeans.log
> >
> > - Web server statistics are available through a very detailed request
> > to infra. They say they're not in the "counting business" :-)
> >
> > - Jan is working in NBM generation (with NOTICE & LICENSE, etc.).
> > Jenkins now generates these artifacts:
> > https://builds.apache.org/view/Incubator%20Projects/job/incubator-netbeans-linux/lastSuccessfulBuild/artifact/nbbuild/nbms/
> >
> >
> > == Open issues & questions
> >
> > 1- It seems we won't be able to host some OSGi bundles in the mirror
> > system due to licensing issues?
> >
> > 2- Mirrors are not as reliable as a CDN: files can be corrupt, or
> > whatever. The UC module should verify the integrity of the downloaded
> > files, etc.
> >
> > 3- We may want to select a small set of NBM files for the first run,
> > to workaround previous licensing problems.
> >
> >
> > [1]
> > https://issues.apache.org/jira/browse/NETBEANS-330
> >
> > [2]
> > https://reference.apache.org/pmc/mirror_scripts
> >
> >
> > On 05/04/18 14:38, Geertjan Wielenga wrote:
> >> Hi all,
> >>
> >> We need to nail down this one and I think the key blocker is that
> >> there are
> >> different ideas about what this is about:
> >>
> >> https://issues.apache.org/jira/browse/NETBEANS-330
> >>
> >> The above is not about the Plugin Portal.
> >>
> >> If I understand it correctly, this is about where the NBMs (which
> >> ones? how
> >> many? do we know?) and the related XML file (a.k.a. update center)
> >> will be
> >> hosted.
> >>
> >> AFAIK, the XML file and the NBMs could be put onto our Apache
> >> NetBeans VM
> >> just like Synergy:
> >>
> >> http://netbeans-vm.apache.org/synergy
> >>
> >> The key question remains, which NBMs are we talking about here,
> >> applicable
> >> to the 9.0 release, I think.
> >>
> >> Thanks,
> >>
> >> Gj
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
> > For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
> >
> > For further information about the NetBeans mailing lists, visit:
> > https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
> >
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Resurrect: Summary of How/where exactly to host the NBMs for the 9.0
release
Posted by Laszlo Kishalmi <la...@gmail.com>.
Dear all,
I have to bring up this old item again. According to the Apache release
processes, I'd need to archive the current 9.0 release from the site.
this means that our 9.0 NBM-s are going to be removed from there as
well. The files are going to be available on archive.apache.org which is
not mirrored. Only the most recent release are supposed to be out on the
release and their mirrors.
This means we need to find some place to our 9.0 nbms. I also won't mind
if we could move away storing our nbms, "officially" (there are no
official binary releases, yet) on some other place than Apache dist
infrastructure. It's a bit weird to sign and upload 650+ binaries with
each and every release. So let's go back to square one.
1. We can change the redirection to the Apache archive site. That is
not mirrored, and initially we would put a real traffic on there.
2. We can serve the nbm-s from our netbeans-vm.
Checked the stats we get ~25000 requests a month on updates.xml.gz
if we'd transfer all the update center (~120 Mb, for NetBeans 10.0)
all the modules with each request for updates.xml.gz our transfer
would be ~3 Tb. So we would definitely fit into the 3-5 Tb/per month
usage limit. That would be actually much less, as this is a worst
case scenario.
3. Search for a free third party solution.
I'd try Bintray, we can get 10 Gb space and CDN, so it is pretty
reliable, the only concern is the 1 Tb of network limit per month.
Unfortunately I only can guess that the 3 Tb is really exaggerated.
All the log files I've found do not include the actual download
statistics of our nbm-s
4. Something else?
I can't archive the current 9.0 till this thing is decided.
On 4/10/18 11:39 PM, Antonio wrote:
> Hi all,
>
> In order to understand this long thread (about NETBEANS-330 [1]) I
> tried to summarize it. Please review and send corrections & questions
> as appropriate.
>
> Kind regards,
> Antonio
>
> == Objectives
>
> Host the NetBeans 9.0 Update Center on Apache infrastructure.
>
> == Constraints/facts/options
>
> - We cannot host it on a website due to bandwidth requirements of 3-5
> Tb/month.
>
> - We must use the Apache Mirror system and their "closer.cgi,
> closer.lua" cgi scripts [2] to select the closest mirror to the user.
> This script can either redirect directly to the closest binary file
> (ready to download), or return a JSON response with a list of closest
> mirrors.
>
> - Distributing through the Apache Mirror system requires a proper
> release, with voting, approval and signing, etc.
>
> - Geertjan says that prior to the final release we could do an rc
> release to have this feature tested.
>
> - The UC catalog xml file can be hosted in the mirror system. For this
> to work the catalog xml file must contain relative paths, so when aged
> releases are moved away from the mirror system and into the archive
> things keep working.
>
> - We can HTTP redirect to the proper catalog.xml file...
>
> a) ... From our website, redirecting to the Apache mirror system or to
> the archive with an .htaccess file (under our control).
>
> b) ... Idem, by asking infra to modify our server configuration (more
> performant but requires Infra tickets).
>
> c) ... Using a script of ours hosted at the "VM" (a web server ours
> currently hosting selenium), that may also track/log some statistics.
>
> - The NetBeans UC module will make a request for the XML file to our
> website, from there will be redirected to the mirror/archive. Later on
> it will build up proper URLs (using some prefix and the relative paths
> inside the catalog xml file) for the final NBM downloads.
>
> - Mirror download statistics may be currently downloaded from
> https://www.apache.org/dyn/stats/netbeans.log
>
> - Web server statistics are available through a very detailed request
> to infra. They say they're not in the "counting business" :-)
>
> - Jan is working in NBM generation (with NOTICE & LICENSE, etc.).
> Jenkins now generates these artifacts:
> https://builds.apache.org/view/Incubator%20Projects/job/incubator-netbeans-linux/lastSuccessfulBuild/artifact/nbbuild/nbms/
>
>
> == Open issues & questions
>
> 1- It seems we won't be able to host some OSGi bundles in the mirror
> system due to licensing issues?
>
> 2- Mirrors are not as reliable as a CDN: files can be corrupt, or
> whatever. The UC module should verify the integrity of the downloaded
> files, etc.
>
> 3- We may want to select a small set of NBM files for the first run,
> to workaround previous licensing problems.
>
>
> [1]
> https://issues.apache.org/jira/browse/NETBEANS-330
>
> [2]
> https://reference.apache.org/pmc/mirror_scripts
>
>
> On 05/04/18 14:38, Geertjan Wielenga wrote:
>> Hi all,
>>
>> We need to nail down this one and I think the key blocker is that
>> there are
>> different ideas about what this is about:
>>
>> https://issues.apache.org/jira/browse/NETBEANS-330
>>
>> The above is not about the Plugin Portal.
>>
>> If I understand it correctly, this is about where the NBMs (which
>> ones? how
>> many? do we know?) and the related XML file (a.k.a. update center)
>> will be
>> hosted.
>>
>> AFAIK, the XML file and the NBMs could be put onto our Apache
>> NetBeans VM
>> just like Synergy:
>>
>> http://netbeans-vm.apache.org/synergy
>>
>> The key question remains, which NBMs are we talking about here,
>> applicable
>> to the 9.0 release, I think.
>>
>> Thanks,
>>
>> Gj
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
> For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
Re: Summary of How/where exactly to host the NBMs for the 9.0 release
Posted by Jan Lahoda <la...@gmail.com>.
On Sat, Apr 14, 2018 at 8:57 AM, Emilian Bold <em...@protonmail.ch>
wrote:
> Ah, so it's a matter of repackaging them?
>
Yes, the (my) current idea for OSGi bundles is to wrap them into NBMs with
LICENSE, etc. This is when creating an update center for the NB
distribution, others may still produce update centers with plain OSGi
wrappers, and it should still work as before.
Jan
> --emi
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>
> On 14 April 2018 9:54 AM, Antonio <an...@vieiro.net> wrote:
>
> > On 14/04/18 08:27, Emilian Bold wrote:
> >
> > > > 1- It seems we won't be able to host some OSGi bundles in the mirror
> > > >
> > > > system due to licensing issues?
> > >
> > > If we are able to make a release with this dependency how are we not
> allowed to distribute the binaries?
> >
> > As Jan said 2018/04/10 (13:08), quoting:
> >
> > "One open issue is what to do with (third-party) OSGi bundles - these
> >
> > are not currently wrapped into NBMs, and it is unclear to me if we can
> >
> > release them as ordinary jars (given these are "upstream" jars which
> >
> > don't have LICENSE, etc.)"
> >
> > Cheers,
> >
> > Antonio
> >
> >
> > ------------------------------------------------------------
> ------------------------------------------------------------
> ------------------------------------------------------------
> ------------------------------------------------------------
> ---------------------------------------------------------------
> >
> > To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
> >
> > For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
> >
> > For further information about the NetBeans mailing lists, visit:
> >
> > https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
> For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
>
Re: Summary of How/where exactly to host the NBMs for the 9.0 release
Posted by Antonio <an...@vieiro.net>.
AFAIK Jan is currently working on that:
https://github.com/apache/incubator-netbeans/pull/494
Un abrazo,
Antonio
P.S.: Current generated artifacts are at:
https://builds.apache.org/view/Incubator%20Projects/job/incubator-netbeans-linux/lastSuccessfulBuild/artifact/nbbuild/nbms/
On 14/04/18 08:57, Emilian Bold wrote:
> Ah, so it's a matter of repackaging them?
>
> --emi
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>
> On 14 April 2018 9:54 AM, Antonio <an...@vieiro.net> wrote:
>
>> On 14/04/18 08:27, Emilian Bold wrote:
>>
>>>> 1- It seems we won't be able to host some OSGi bundles in the mirror
>>>>
>>>> system due to licensing issues?
>>>
>>> If we are able to make a release with this dependency how are we not allowed to distribute the binaries?
>>
>> As Jan said 2018/04/10 (13:08), quoting:
>>
>> "One open issue is what to do with (third-party) OSGi bundles - these
>>
>> are not currently wrapped into NBMs, and it is unclear to me if we can
>>
>> release them as ordinary jars (given these are "upstream" jars which
>>
>> don't have LICENSE, etc.)"
>>
>> Cheers,
>>
>> Antonio
>>
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
>>
>> For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
>>
>> For further information about the NetBeans mailing lists, visit:
>>
>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
> For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Summary of How/where exactly to host the NBMs for the 9.0 release
Posted by Emilian Bold <em...@protonmail.ch>.
Ah, so it's a matter of repackaging them?
--emi
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On 14 April 2018 9:54 AM, Antonio <an...@vieiro.net> wrote:
> On 14/04/18 08:27, Emilian Bold wrote:
>
> > > 1- It seems we won't be able to host some OSGi bundles in the mirror
> > >
> > > system due to licensing issues?
> >
> > If we are able to make a release with this dependency how are we not allowed to distribute the binaries?
>
> As Jan said 2018/04/10 (13:08), quoting:
>
> "One open issue is what to do with (third-party) OSGi bundles - these
>
> are not currently wrapped into NBMs, and it is unclear to me if we can
>
> release them as ordinary jars (given these are "upstream" jars which
>
> don't have LICENSE, etc.)"
>
> Cheers,
>
> Antonio
>
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
>
> For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
>
> For further information about the NetBeans mailing lists, visit:
>
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Summary of How/where exactly to host the NBMs for the 9.0 release
Posted by Antonio <an...@vieiro.net>.
On 14/04/18 08:27, Emilian Bold wrote:
>> 1- It seems we won't be able to host some OSGi bundles in the mirror
> system due to licensing issues?
>
> If we are able to make a release with this dependency how are we not allowed to distribute the binaries?
>
As Jan said 2018/04/10 (13:08), quoting:
"One open issue is what to do with (third-party) OSGi bundles - these
are not currently wrapped into NBMs, and it is unclear to me if we can
release them as ordinary jars (given these are "upstream" jars which
don't have LICENSE, etc.)"
Cheers,
Antonio
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Summary of How/where exactly to host the NBMs for the 9.0 release
Posted by Emilian Bold <em...@protonmail.ch>.
> 1- It seems we won't be able to host some OSGi bundles in the mirror
system due to licensing issues?
If we are able to make a release with this dependency how are we not allowed to distribute the binaries?
--emi
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On 11 April 2018 9:39 AM, Antonio <an...@vieiro.net> wrote:
> Hi all,
>
> In order to understand this long thread (about NETBEANS-330 [1]) I tried
>
> to summarize it. Please review and send corrections & questions as
>
> appropriate.
>
> Kind regards,
>
> Antonio
>
> == Objectives
>
> Host the NetBeans 9.0 Update Center on Apache infrastructure.
>
> == Constraints/facts/options
>
> - We cannot host it on a website due to bandwidth requirements of 3-5
>
> Tb/month.
>
> - We must use the Apache Mirror system and their "closer.cgi,
>
> closer.lua" cgi scripts [2] to select the closest mirror to the user.
>
> This script can either redirect directly to the closest binary file
>
> (ready to download), or return a JSON response with a list of closest
>
> mirrors.
>
> - Distributing through the Apache Mirror system requires a proper
>
> release, with voting, approval and signing, etc.
>
> - Geertjan says that prior to the final release we could do an rc
>
> release to have this feature tested.
>
> - The UC catalog xml file can be hosted in the mirror system. For this
>
> to work the catalog xml file must contain relative paths, so when aged
>
> releases are moved away from the mirror system and into the archive
>
> things keep working.
>
> - We can HTTP redirect to the proper catalog.xml file...
>
> a) ... From our website, redirecting to the Apache mirror system or to
>
> the archive with an .htaccess file (under our control).
>
> b) ... Idem, by asking infra to modify our server configuration (more
>
> performant but requires Infra tickets).
>
> c) ... Using a script of ours hosted at the "VM" (a web server ours
>
> currently hosting selenium), that may also track/log some statistics.
>
> - The NetBeans UC module will make a request for the XML file to our
>
> website, from there will be redirected to the mirror/archive. Later on
>
> it will build up proper URLs (using some prefix and the relative paths
>
> inside the catalog xml file) for the final NBM downloads.
>
> - Mirror download statistics may be currently downloaded from
>
> https://www.apache.org/dyn/stats/netbeans.log
>
> - Web server statistics are available through a very detailed request to
>
> infra. They say they're not in the "counting business" :-)
>
> - Jan is working in NBM generation (with NOTICE & LICENSE, etc.).
>
> Jenkins now generates these artifacts:
>
> https://builds.apache.org/view/Incubator Projects/job/incubator-netbeans-linux/lastSuccessfulBuild/artifact/nbbuild/nbms/
>
> == Open issues & questions
>
> 1- It seems we won't be able to host some OSGi bundles in the mirror
>
> system due to licensing issues?
>
> 2- Mirrors are not as reliable as a CDN: files can be corrupt, or
>
> whatever. The UC module should verify the integrity of the downloaded
>
> files, etc.
>
> 3- We may want to select a small set of NBM files for the first run, to
>
> workaround previous licensing problems.
>
> [1]
>
> https://issues.apache.org/jira/browse/NETBEANS-330
>
> [2]
>
> https://reference.apache.org/pmc/mirror_scripts
>
> On 05/04/18 14:38, Geertjan Wielenga wrote:
>
>
> > Hi all,
> >
> > We need to nail down this one and I think the key blocker is that there are
> >
> > different ideas about what this is about:
> >
> > https://issues.apache.org/jira/browse/NETBEANS-330
> >
> > The above is not about the Plugin Portal.
> >
> > If I understand it correctly, this is about where the NBMs (which ones? how
> >
> > many? do we know?) and the related XML file (a.k.a. update center) will be
> >
> > hosted.
> >
> > AFAIK, the XML file and the NBMs could be put onto our Apache NetBeans VM
> >
> > just like Synergy:
> >
> > http://netbeans-vm.apache.org/synergy
> >
> > The key question remains, which NBMs are we talking about here, applicable
> >
> > to the 9.0 release, I think.
> >
> > Thanks,
> >
> > Gj
>
> --
>
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
>
> For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
>
> For further information about the NetBeans mailing lists, visit:
>
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.incubator.apache.org
For additional commands, e-mail: dev-help@netbeans.incubator.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists