You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@netbeans.apache.org by antonio <an...@vieiro.net> on 2021/03/21 20:04:59 UTC
Compiling shared libraries (cnd.remote)
Hi all,
As you may know the module "cnd.remote" uses some shared libraries to
access files through ssh [1]. It seems the source code for these shared
libraries was included in the 4th donation (see [2]).
I've asked Infra if we could have Jenkins agents for compiling on
Solaris and MacOS, but it seems we only have Windows and Linux Jenkins
buildbots.
Questions are:
1) Do we want to drop Solaris support? If not, would it be possible to
get an Oracle box/docker image for compiling the library? Once compiled
I don't think we'll have to compile it again frequently.
2) ASF Infra doesn't have MacOS agents, but maybe we can get one in
Azure (they're finding out). How are we currently compiling the MacOS
installers?
Thanks,
Antonio
[1]
https://github.com/apache/netbeans/blob/bf55111bb537c857f70d3fbaa80f04481cb2eb87/cnd/cnd.remote/nbproject/project.properties#L30
We have previous binary versions of the shared libraries (CDDL License)
for Solaris and Linux, but there're different fingerprints:
https://netbeans.osuosl.org/binaries/2B7152611CB7729C6B5E61E7E2E02060D56FFE1D-cnd-rfs-1.0.zip
and here:
https://netbeans.osuosl.org/binaries/2BCF2047382FB68A2F275677745C80E79B4046AB-cnd-rfs-1.0.zip
[2]
https://github.com/apache/netbeans/tree/cnd/cnd/cnd.remote/tools
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Compiling shared libraries (cnd.remote)
Posted by antonio <an...@vieiro.net>.
Some comments below.
El 2/4/21 a las 19:06, Lars Bruun-Hansen escribió:
> Hi Antonio
>
> missed this one.
>
> Specially for cnd.remote when you ask what platforms should be
> supported, do you mean where the IDE is running or the platform OS of
> the remote target?
As far as I understand:
- ide/dlight.nativeexecution runs locally (note this has _many_ shared
libraries as well which we don't currently compile) and does indeed
depend on JSch
https://github.com/apache/netbeans/tree/master/ide/dlight.nativeexecution/tools
- cnd/cnd.remote is copied to the remote server using this script:
https://github.com/apache/netbeans/blob/cnd/cnd/cnd.remote/tools/upload_rfs_binaries.bash
it is a server-side server with a specific protocol that sends
information about available remote tooling (compilers, etc.) and
filesystem information back to the IDE.
So you can run the IDE in a Linux box and build on a Solaris/Linux
remote box, for instance.
> Example: Solaris hasn't been relevant as a desktop OS for decades but
> is (was?) still relevant as a remote development target for
> cnd.remote.
I think Oracle still sells Oracle Studio IDE, which is based on NetBeans
cnd, and runs on Solaris and Linux.
I imagine people developing for Solaris and OpenSolaris derivatives have
Emacs and Makefiles for breakfast, compile assembly by hand for lunch
and don't use NetBeans :-).
>
> If you mean where the IDE is running then I think we should _only_
> support the following platforms going forward:
>
> - Linux, 32-bit, X86
> - Linux, 64-bit, X64
> - MacOS, 64-bit, X64
> - Windows, 32-bit, X86
> - Windows, 64-bit, X64
D'oh! I'm a FreeBSD fan :-). Luckily FreeBSD runs Linux native binaries!
The list looks sensible to me.
>
> Note 1: We'll probably soon be forced to support Apple Silicon too
> (fancy marketing word for MacOS on ARM-64), but as far as I understand
> it is possible to build these binaries on X64)
"We'll probably soon be forced to support..." [add closed/undocumented
platform here] is not on my open source book :-).
> Note 2: I for one would like to drop support for 32-bit platforms, but
> I don't think we are all there?
Can't tell. I ran a FreeBSD-32bit several years ago, have been on 64bit
for ages now.
>
> For the NB lib.profiler the same issue was faced. In the past some
> esoteric platforms like Solaris on SPARC and HP-UX on PA-RISC had been
> supported and there were pre-build binaries for those in the NetBeans
> distro. But now there was a need for a bug fix. The solution was to
> create new binaries only for the above platforms (using GitHub
> Actions) and then overlay those onto the older binaries. This way the
> older binaries are still there but they'll no longer receive bug
> fixes. This may be a solution for cnd.remote too ?
Yes, I think so. But note that there're quite a few shared libraries in
NetBeans, and maybe a single yml file is not good enough for all of them.
I don't think we're required to compile binaries on ASF Infra, but maybe
we are. We should investigate.
>
> On another note: Native binaries are not that easy to tackle for the
> NetBeans community. They should exist for a good reason. So another
> question which one may ask for cnd.remote: Can't the same be
> accomplished these days by using a Java native solution? I've used
> JSch for this purpose in the past and it works well. But there may be
> many pros and cons I can't see.
>
I think the best plan with CND could be something like this:
1.- Clean up all licenses.
2.- Compile it (possibly adding missing files).
3.- Run it (we'll face wrong paths due to cluster reorganization)
4.- Test it.
5.- After that we can see what features we want to support, or change.
Who knows, maybe CND and LSP work well together.
Cheers,
Antonio
P.S.: For more details about remote development see:
https://netbeans.apache.org/kb/docs/cnd/remote-modes.html
https://netbeans.apache.org/kb/docs/cnd/remotedev-tutorial.html
>
> /Lars
>
>
>
>
>
>
>
>
>
>
>
> On Fri, Apr 2, 2021 at 5:53 PM antonio <an...@vieiro.net> wrote:
>>
>> Some comments below
>>
>> El 2/4/21 a las 17:12, Christian Oyarzun escribió:
>>> Hi Antonio,
>>>
>>> 1) +1 for dropping Solaris.
>>
>> I managed to download a Solaris Virtual Box image (3Gb) and install it
>> on my box (with a free subscription for myself). This may be good for
>> compiling for Solaris 11 X64, but not for Sparc. Haven't tried this path
>> yet, though.
>>
>> It would be great if Oracle donated some hardware/cloud agent to the ASF
>> for cross-compilation in Solaris. This could be used by different Apache
>> projects also. Any Oraclers want to push this internally? General
>> instructions on how to host an external agent are here:
>>
>> https://infra.apache.org/hosting-external-agent.html
>>
>>>
>>> 2) Have you heard back from ASF infra in regard to macOS agents?
>>
>> They're waiting for us to request them. AFAIU the request may or may not
>> go ahead (depending on budget and mainteinance costs, I imagine).
>>
>>>
>>> Since AWS now has macOS EC2 instances, maybe we could apply for promotional
>>> credits for a Jenkins agent?
>>> https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/
>>> https://aws.amazon.com/ec2/instance-types/mac/
>>> https://issues.apache.org/jira/browse/INFRA-17961
>>
>> Azure has also macOS agents, but for Azure pipelines, not Jenkins.
>>
>> https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml
>>
>> List of tooling is impressive:
>>
>> https://github.com/actions/virtual-environments/blob/main/images/macos/macos-10.14-Readme.md#java
>>
>> Shall I reopen INFRA-17961? If so we may want to have a list of things
>> we want to do with these agents. Things I can think of:
>>
>> - Build the NetBeans shared libraries on MacOS.
>> - Build some CND shared libraries for MacOS.
>> - Build the installers? Are these APL-v2 licensed?
>> - Anything else?
>>
>> Kind regards,
>> Antonio
>>
>> P.S.: Both AWS and Azure are ASF platinum sponsors!
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
>> For additional commands, e-mail: dev-help@netbeans.apache.org
>>
>> For further information about the NetBeans mailing lists, visit:
>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Compiling shared libraries (cnd.remote)
Posted by Lars Bruun-Hansen <lb...@gmail.com>.
Hi Antonio
missed this one.
Specially for cnd.remote when you ask what platforms should be
supported, do you mean where the IDE is running or the platform OS of
the remote target?
Example: Solaris hasn't been relevant as a desktop OS for decades but
is (was?) still relevant as a remote development target for
cnd.remote.
If you mean where the IDE is running then I think we should _only_
support the following platforms going forward:
- Linux, 32-bit, X86
- Linux, 64-bit, X64
- MacOS, 64-bit, X64
- Windows, 32-bit, X86
- Windows, 64-bit, X64
Note 1: We'll probably soon be forced to support Apple Silicon too
(fancy marketing word for MacOS on ARM-64), but as far as I understand
it is possible to build these binaries on X64)
Note 2: I for one would like to drop support for 32-bit platforms, but
I don't think we are all there?
For the NB lib.profiler the same issue was faced. In the past some
esoteric platforms like Solaris on SPARC and HP-UX on PA-RISC had been
supported and there were pre-build binaries for those in the NetBeans
distro. But now there was a need for a bug fix. The solution was to
create new binaries only for the above platforms (using GitHub
Actions) and then overlay those onto the older binaries. This way the
older binaries are still there but they'll no longer receive bug
fixes. This may be a solution for cnd.remote too ?
On another note: Native binaries are not that easy to tackle for the
NetBeans community. They should exist for a good reason. So another
question which one may ask for cnd.remote: Can't the same be
accomplished these days by using a Java native solution? I've used
JSch for this purpose in the past and it works well. But there may be
many pros and cons I can't see.
/Lars
On Fri, Apr 2, 2021 at 5:53 PM antonio <an...@vieiro.net> wrote:
>
> Some comments below
>
> El 2/4/21 a las 17:12, Christian Oyarzun escribió:
> > Hi Antonio,
> >
> > 1) +1 for dropping Solaris.
>
> I managed to download a Solaris Virtual Box image (3Gb) and install it
> on my box (with a free subscription for myself). This may be good for
> compiling for Solaris 11 X64, but not for Sparc. Haven't tried this path
> yet, though.
>
> It would be great if Oracle donated some hardware/cloud agent to the ASF
> for cross-compilation in Solaris. This could be used by different Apache
> projects also. Any Oraclers want to push this internally? General
> instructions on how to host an external agent are here:
>
> https://infra.apache.org/hosting-external-agent.html
>
> >
> > 2) Have you heard back from ASF infra in regard to macOS agents?
>
> They're waiting for us to request them. AFAIU the request may or may not
> go ahead (depending on budget and mainteinance costs, I imagine).
>
> >
> > Since AWS now has macOS EC2 instances, maybe we could apply for promotional
> > credits for a Jenkins agent?
> > https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/
> > https://aws.amazon.com/ec2/instance-types/mac/
> > https://issues.apache.org/jira/browse/INFRA-17961
>
> Azure has also macOS agents, but for Azure pipelines, not Jenkins.
>
> https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml
>
> List of tooling is impressive:
>
> https://github.com/actions/virtual-environments/blob/main/images/macos/macos-10.14-Readme.md#java
>
> Shall I reopen INFRA-17961? If so we may want to have a list of things
> we want to do with these agents. Things I can think of:
>
> - Build the NetBeans shared libraries on MacOS.
> - Build some CND shared libraries for MacOS.
> - Build the installers? Are these APL-v2 licensed?
> - Anything else?
>
> Kind regards,
> Antonio
>
> P.S.: Both AWS and Azure are ASF platinum sponsors!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Compiling shared libraries (cnd.remote)
Posted by antonio <an...@vieiro.net>.
Some comments below
El 2/4/21 a las 17:12, Christian Oyarzun escribió:
> Hi Antonio,
>
> 1) +1 for dropping Solaris.
I managed to download a Solaris Virtual Box image (3Gb) and install it
on my box (with a free subscription for myself). This may be good for
compiling for Solaris 11 X64, but not for Sparc. Haven't tried this path
yet, though.
It would be great if Oracle donated some hardware/cloud agent to the ASF
for cross-compilation in Solaris. This could be used by different Apache
projects also. Any Oraclers want to push this internally? General
instructions on how to host an external agent are here:
https://infra.apache.org/hosting-external-agent.html
>
> 2) Have you heard back from ASF infra in regard to macOS agents?
They're waiting for us to request them. AFAIU the request may or may not
go ahead (depending on budget and mainteinance costs, I imagine).
>
> Since AWS now has macOS EC2 instances, maybe we could apply for promotional
> credits for a Jenkins agent?
> https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/
> https://aws.amazon.com/ec2/instance-types/mac/
> https://issues.apache.org/jira/browse/INFRA-17961
Azure has also macOS agents, but for Azure pipelines, not Jenkins.
https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml
List of tooling is impressive:
https://github.com/actions/virtual-environments/blob/main/images/macos/macos-10.14-Readme.md#java
Shall I reopen INFRA-17961? If so we may want to have a list of things
we want to do with these agents. Things I can think of:
- Build the NetBeans shared libraries on MacOS.
- Build some CND shared libraries for MacOS.
- Build the installers? Are these APL-v2 licensed?
- Anything else?
Kind regards,
Antonio
P.S.: Both AWS and Azure are ASF platinum sponsors!
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Compiling shared libraries (cnd.remote)
Posted by Christian Oyarzun <co...@oyarzun.net>.
Hi Antonio,
1) +1 for dropping Solaris.
2) Have you heard back from ASF infra in regard to macOS agents?
Since AWS now has macOS EC2 instances, maybe we could apply for promotional
credits for a Jenkins agent?
https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/
https://aws.amazon.com/ec2/instance-types/mac/
https://issues.apache.org/jira/browse/INFRA-17961
Best regards,
Christian
On Sun, Mar 21, 2021 at 4:05 PM antonio <an...@vieiro.net> wrote:
> Hi all,
>
> As you may know the module "cnd.remote" uses some shared libraries to
> access files through ssh [1]. It seems the source code for these shared
> libraries was included in the 4th donation (see [2]).
>
> I've asked Infra if we could have Jenkins agents for compiling on
> Solaris and MacOS, but it seems we only have Windows and Linux Jenkins
> buildbots.
>
> Questions are:
>
> 1) Do we want to drop Solaris support? If not, would it be possible to
> get an Oracle box/docker image for compiling the library? Once compiled
> I don't think we'll have to compile it again frequently.
>
> 2) ASF Infra doesn't have MacOS agents, but maybe we can get one in
> Azure (they're finding out). How are we currently compiling the MacOS
> installers?
>
> Thanks,
> Antonio
>
>
> [1]
>
> https://github.com/apache/netbeans/blob/bf55111bb537c857f70d3fbaa80f04481cb2eb87/cnd/cnd.remote/nbproject/project.properties#L30
>
> We have previous binary versions of the shared libraries (CDDL License)
> for Solaris and Linux, but there're different fingerprints:
>
>
> https://netbeans.osuosl.org/binaries/2B7152611CB7729C6B5E61E7E2E02060D56FFE1D-cnd-rfs-1.0.zip
> and here:
>
> https://netbeans.osuosl.org/binaries/2BCF2047382FB68A2F275677745C80E79B4046AB-cnd-rfs-1.0.zip
>
> [2]
> https://github.com/apache/netbeans/tree/cnd/cnd/cnd.remote/tools
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
>