You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Alessio Gottardo <ta...@yahoo.it.INVALID> on 2017/11/20 17:11:01 UTC

Compiling proton-c for Linux Alpine

Hi there,
I am trying to compile Qpid Proton on Linux Alpine (within a docker image).
I am getting this error:
/go/src/qpid-proton/proton-c/src/proactor/epoll.c: In function 'pmutex_init':/go/src/qpid-proton/proton-c/src/proactor/epoll.c:104:36: error: 'PTHREAD_MUTEX_ADAPTIVE_NP' undeclared (first use in this function)   pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_ADAPTIVE_NP);                                    ^~~~~~~~~~~~~~~~~~~~~~~~~/go/src/qpid-proton/proton-c/src/proactor/epoll.c:104:36: note: each undeclared identifier is reported only once for each function it appears inmake[2]: *** [proton-c/CMakeFiles/qpid-proton.dir/build.make:695: proton-c/CMakeFiles/qpid-proton.dir/src/proactor/epoll.c.o] Error 1make[1]: *** [CMakeFiles/Makefile2:1069: proton-c/CMakeFiles/qpid-proton.dir/all] Error 2make: *** [Makefile:141: all] Error 2
The relevant part of the Dockerfile (FROM golang:1.8.3-alpine3.6):
```# download Apache QPIDWORKDIR /go/src/RUN git clone --progress --verbose http://git.apache.org/qpid-proton.git
# dependencies for Apache QPID (check https://pkgs.alpinelinux.org/packages)RUN apk add gcc=6.3.0-r4 g++=6.3.0-r4 cmake=3.8.1-r0 make=4.2.1-r0RUN apk add python2=2.7.13-r1RUN apk add libssl1.0=1.0.2m-r0 cyrus-sasl=2.1.26-r10RUN apk add swig=3.0.10-r0
# compile and install Apache QPIDWORKDIR /go/src/qpid-protonRUN git checkout tags/0.18.0WORKDIR /go/src/qpid-proton/buildRUN cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DSYSINSTALL_BINDINGS=ONRUN make install```
The previous steps are based on the details I found here: https://github.com/apache/qpid-proton/blob/master/INSTALL.md
The reason for doing this is to be able to run Go applications using Docker and the Electron library for QPID.
I am not familiar with compiling C projects and I am not able to proceed based on that error message.It seems some header files are missing or something like that but I can not figure out which dependency is missing.Any hint or help on how to compile this library for Linux Alpine?
Thank youAlessio


Re: Compiling proton-c for Linux Alpine

Posted by Alan Conway <ac...@redhat.com>.
On Thu, Nov 23, 2017 at 5:48 AM, Chris Richardson <cr...@fourc.eu> wrote:

> [snip: discussion of issues building on Linux Alpine]
>
> Another alternative is that we make a more official version of the source
> code hack I previously mentioned to make Proton compatible with the reduced
> (I'm not judging) musl libc. Any thoughts, Messrs. Stitcher, Conway et al?
>
>
In general I like the idea of making the core more portable and reducing
dependencies, of course the devil is in the details. If you can put forward
something that works for you as a JIRA  we can figure out how/if to adopt
the changes into the core.

I haven't followed the ins and outs but I have a couple comments:

> > tested it at all, but if glibc isn't an option you could change
> > the PTHREAD_MUTEX_ADAPTIVE_NP to PTHREAD_MUTEX_NORMAL. I _think_ the only
> > side effect would be that some sub-optimal system calls


I suspect we can produce a more portable version of epoll.c that uses the
more modern pthread stuff conditionally (or not at all) - Cliff is the
expert.

Another option is to use a different proactor - although of you have epoll
then making the epoll proactor work is probably the best option. We have
been considering whether or not we need a plain Unix poll() or select()
proactor but we are hoping the libuv proactor will fill that gap. I don't
know if libuv is an option for you, but since you want less dependencies
rather than more it probably isn't.

Cheers,
Alan.

Re: Compiling proton-c for Linux Alpine

Posted by Chris Richardson <cr...@fourc.eu>.
On 21 November 2017 at 17:41, Alessio Gottardo <ta...@yahoo.it.invalid>
wrote:

>  Thank you Chris,
>
> I've been reading about this new world (to me) of the reasons behind glibc
> and ulibc. In some way there are similar issues in "toy languages / higher
> level languages" like java or javascript (node.js) when shipping the
> equivalent of executables with tons of dependencies (and transitive
> dependencies) making the artefact huge in terms of disk space.
>
> Anyway, I've added the following to the docker file (using RUN statements)
> in order to include the binary package of glibc to Linux Alpine (cf.
> https://github.com/sgerrand/alpine-pkg-glibc#installing):
>
>
> apk --no-cache add ca-certificates wget
> wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://raw.githubusercontent.
> com/sgerrand/alpine-pkg-glibc/master/sgerrand.rsa.pub
> wget https://github.com/sgerrand/alpine-pkg-glibc/releases/
> download/2.25-r0/glibc-2.25-r0.apk
> apk add glibc-2.25-r0.apk
>
> However I get the same compilation error.
>
> I believe that's because somehow I need to point the cmake/make stuff to
> glibc (instead of the default/system ulibc) hardcoding somewhere the path?
> If that's the case then I am not sure where I should change this stuff and
> which path I should add.
> Please let me know if this rings any bell.
>

I've followed these installation steps and now realise that the glibc
package above is runtime only and does not support local compilation - were
it to do so it would probably be called "glibc-dev" by convention. This
means that you can only run packages that were compiled elsewhere, which is
presumably the case for JDKs etc. In retrospect an obvious choice when the
target is a minimalist OS so apologies for leading you down a bit of a
blind alley.

In terms of progressing further I would suggest you might look into
extending a build system such as sgerrand's (the one that generates the
glibc.apk package) in order to either build a glibc-dev package (this will
probably result in a bloated Alpine image) or to build qpid-proton and
export it as a .apk file. The latter might be the most satisfactory
solution since you would then not require many of the build dependencies
you already have (git, cmake, gcc etc) in order to install qpid-proton in
the live environment. The down-side is that it's quite and involved process
and certainly not one for the faint of heart!

Another alternative is that we make a more official version of the source
code hack I previously mentioned to make Proton compatible with the reduced
(I'm not judging) musl libc. Any thoughts, Messrs. Stitcher, Conway et al?

Thanks
> Alessio
>
>
>
>
>
>     On Monday, 20 November 2017, 19:34:50 GMT, Chris Richardson <
> cr@fourc.eu> wrote:
>
>  Hi Alessio,
> The problem here appears to be that qpid-proton assumes glibc is installed
> on your system (it is on "most"(tm) linuces), but alpine doesn't appear to
> contain it, nor yet is it available in the default apk repositories. The
> pthread-stub package which provides /usr/include/pthread.h does not include
> the expected PTHREAD_MUTEX_ADAPTIVE_NP definition.
> The easiest option might be to use a docker image which does include
> glibc, try one of the documented solutions here http://wiki.alpinelinux.
> org/wiki/Running_glibc_programs or here https://github.com/sgerrand.
> This is a bit of a hack and comes with a huge disclaimer as I haven't
> tested it at all, but if glibc isn't an option you could change
> the PTHREAD_MUTEX_ADAPTIVE_NP to PTHREAD_MUTEX_NORMAL. I _think_ the only
> side effect would be that some sub-optimal system calls might take place so
> the thread locking would not be as efficient.I've attached a patch to do
> this which you can apply withpatch -p1 < qpid-proton-0.18.0-pthread-
> stub.patch
> I hope this helps to point you in the right direction.
> Chris
>
> On 20 November 2017 at 17:11, Alessio Gottardo <ta...@yahoo.it.invalid>
> wrote:
>
> Hi there,
> I am trying to compile Qpid Proton on Linux Alpine (within a docker image).
> I am getting this error:
> /go/src/qpid-proton/proton-c/ src/proactor/epoll.c: In function
> 'pmutex_init':/go/src/qpid- proton/proton-c/src/proactor/ epoll.c:104:36:
> error: 'PTHREAD_MUTEX_ADAPTIVE_NP' undeclared (first use in this function)
>  pthread_mutexattr_settype(& attr, PTHREAD_MUTEX_ADAPTIVE_NP);
>                       ^~~~~~~~~~~~~~~~~~~~~~~~~/go/
> src/qpid-proton/proton-c/src/ proactor/epoll.c:104:36: note: each
> undeclared identifier is reported only once for each function it appears
> inmake[2]: *** [proton-c/CMakeFiles/qpid- proton.dir/build.make:695:
> proton-c/CMakeFiles/qpid- proton.dir/src/proactor/epoll. c.o] Error
> 1make[1]: *** [CMakeFiles/Makefile2:1069: proton-c/CMakeFiles/qpid-
> proton.dir/all] Error 2make: *** [Makefile:141: all] Error 2
> The relevant part of the Dockerfile (FROM golang:1.8.3-alpine3.6):
> ```# download Apache QPIDWORKDIR /go/src/RUN git clone --progress
> --verbose http://git.apache.org/qpid- proton.git
> # dependencies for Apache QPID (check https://pkgs.alpinelinux.org/
> packages)RUN apk add gcc=6.3.0-r4 g++=6.3.0-r4 cmake=3.8.1-r0
> make=4.2.1-r0RUN apk add python2=2.7.13-r1RUN apk add libssl1.0=1.0.2m-r0
> cyrus-sasl=2.1.26-r10RUN apk add swig=3.0.10-r0
> # compile and install Apache QPIDWORKDIR /go/src/qpid-protonRUN git
> checkout tags/0.18.0WORKDIR /go/src/qpid-proton/buildRUN cmake ..
> -DCMAKE_INSTALL_PREFIX=/usr -DSYSINSTALL_BINDINGS=ONRUN make install```
> The previous steps are based on the details I found here:
> https://github.com/ apache/qpid-proton/blob/ master/INSTALL.md
> The reason for doing this is to be able to run Go applications using
> Docker and the Electron library for QPID.
> I am not familiar with compiling C projects and I am not able to proceed
> based on that error message.It seems some header files are missing or
> something like that but I can not figure out which dependency is
> missing.Any hint or help on how to compile this library for Linux Alpine?
> Thank youAlessio
>
>
>
>
>
> --
>
> Chris Richardson, System Architect
> cr@fourc.eu
>
> FourC AS, Vestre Rosten 81, Trekanten, NO-7075 Tiller, Norway
> www.fourc.eu
>
> Follow us on LinkedIn, Facebook, Google+ and Twitter!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org




-- 

*Chris Richardson*, System Architect
cr@fourc.eu


*FourC AS, Vestre Rosten 81, Trekanten, NO-7075 Tiller, Norwaywww.fourc.eu
<http://www.fourc.eu/>*

*Follow us on LinkedIn <http://bit.ly/fourcli>, Facebook
<http://bit.ly/fourcfb>, Google+ <http://bit.ly/fourcgp> and Twitter
<http://bit.ly/fourctw>!*

Re: Compiling proton-c for Linux Alpine

Posted by Alessio Gottardo <ta...@yahoo.it.INVALID>.
 Thank you Chris,

I've been reading about this new world (to me) of the reasons behind glibc and ulibc. In some way there are similar issues in "toy languages / higher level languages" like java or javascript (node.js) when shipping the equivalent of executables with tons of dependencies (and transitive dependencies) making the artefact huge in terms of disk space.

Anyway, I've added the following to the docker file (using RUN statements) in order to include the binary package of glibc to Linux Alpine (cf. https://github.com/sgerrand/alpine-pkg-glibc#installing):


apk --no-cache add ca-certificates wget
wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://raw.githubusercontent.com/sgerrand/alpine-pkg-glibc/master/sgerrand.rsa.pub
wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.25-r0/glibc-2.25-r0.apk
apk add glibc-2.25-r0.apk

However I get the same compilation error.

I believe that's because somehow I need to point the cmake/make stuff to glibc (instead of the default/system ulibc) hardcoding somewhere the path?
If that's the case then I am not sure where I should change this stuff and which path I should add. 
Please let me know if this rings any bell.
Thanks
Alessio





    On Monday, 20 November 2017, 19:34:50 GMT, Chris Richardson <cr...@fourc.eu> wrote:  
 
 Hi Alessio,
The problem here appears to be that qpid-proton assumes glibc is installed on your system (it is on "most"(tm) linuces), but alpine doesn't appear to contain it, nor yet is it available in the default apk repositories. The pthread-stub package which provides /usr/include/pthread.h does not include the expected PTHREAD_MUTEX_ADAPTIVE_NP definition.
The easiest option might be to use a docker image which does include glibc, try one of the documented solutions here http://wiki.alpinelinux.org/wiki/Running_glibc_programs or here https://github.com/sgerrand.
This is a bit of a hack and comes with a huge disclaimer as I haven't tested it at all, but if glibc isn't an option you could change the PTHREAD_MUTEX_ADAPTIVE_NP to PTHREAD_MUTEX_NORMAL. I _think_ the only side effect would be that some sub-optimal system calls might take place so the thread locking would not be as efficient.I've attached a patch to do this which you can apply withpatch -p1 < qpid-proton-0.18.0-pthread-stub.patch
I hope this helps to point you in the right direction.
Chris

On 20 November 2017 at 17:11, Alessio Gottardo <ta...@yahoo.it.invalid> wrote:

Hi there,
I am trying to compile Qpid Proton on Linux Alpine (within a docker image).
I am getting this error:
/go/src/qpid-proton/proton-c/ src/proactor/epoll.c: In function 'pmutex_init':/go/src/qpid- proton/proton-c/src/proactor/ epoll.c:104:36: error: 'PTHREAD_MUTEX_ADAPTIVE_NP' undeclared (first use in this function)   pthread_mutexattr_settype(& attr, PTHREAD_MUTEX_ADAPTIVE_NP);                                    ^~~~~~~~~~~~~~~~~~~~~~~~~/go/ src/qpid-proton/proton-c/src/ proactor/epoll.c:104:36: note: each undeclared identifier is reported only once for each function it appears inmake[2]: *** [proton-c/CMakeFiles/qpid- proton.dir/build.make:695: proton-c/CMakeFiles/qpid- proton.dir/src/proactor/epoll. c.o] Error 1make[1]: *** [CMakeFiles/Makefile2:1069: proton-c/CMakeFiles/qpid- proton.dir/all] Error 2make: *** [Makefile:141: all] Error 2
The relevant part of the Dockerfile (FROM golang:1.8.3-alpine3.6):
```# download Apache QPIDWORKDIR /go/src/RUN git clone --progress --verbose http://git.apache.org/qpid- proton.git
# dependencies for Apache QPID (check https://pkgs.alpinelinux.org/ packages)RUN apk add gcc=6.3.0-r4 g++=6.3.0-r4 cmake=3.8.1-r0 make=4.2.1-r0RUN apk add python2=2.7.13-r1RUN apk add libssl1.0=1.0.2m-r0 cyrus-sasl=2.1.26-r10RUN apk add swig=3.0.10-r0
# compile and install Apache QPIDWORKDIR /go/src/qpid-protonRUN git checkout tags/0.18.0WORKDIR /go/src/qpid-proton/buildRUN cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DSYSINSTALL_BINDINGS=ONRUN make install```
The previous steps are based on the details I found here: https://github.com/ apache/qpid-proton/blob/ master/INSTALL.md
The reason for doing this is to be able to run Go applications using Docker and the Electron library for QPID.
I am not familiar with compiling C projects and I am not able to proceed based on that error message.It seems some header files are missing or something like that but I can not figure out which dependency is missing.Any hint or help on how to compile this library for Linux Alpine?
Thank youAlessio





-- 

Chris Richardson, System Architect
cr@fourc.eu

FourC AS, Vestre Rosten 81, Trekanten, NO-7075 Tiller, Norway
www.fourc.eu

Follow us on LinkedIn, Facebook, Google+ and Twitter!

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org  

Re: Compiling proton-c for Linux Alpine

Posted by Chris Richardson <cr...@fourc.eu>.
Hi Alessio,

The problem here appears to be that qpid-proton assumes glibc is installed
on your system (it is on "most"(tm) linuces), but alpine doesn't appear to
contain it, nor yet is it available in the default apk repositories. The
pthread-stub package which provides /usr/include/pthread.h does not include
the expected PTHREAD_MUTEX_ADAPTIVE_NP definition.

The easiest option might be to use a docker image which does include glibc,
try one of the documented solutions here
http://wiki.alpinelinux.org/wiki/Running_glibc_programs or here
https://github.com/sgerrand.

This is a bit of a hack and comes with a huge disclaimer as I haven't
tested it at all, but if glibc isn't an option you could change the
PTHREAD_MUTEX_ADAPTIVE_NP
to PTHREAD_MUTEX_NORMAL. I _think_ the only side effect would be that some
sub-optimal system calls might take place so the thread locking would not
be as efficient.
I've attached a patch to do this which you can apply with
patch -p1 < qpid-proton-0.18.0-pthread-stub.patch

I hope this helps to point you in the right direction.

Chris


On 20 November 2017 at 17:11, Alessio Gottardo <ta...@yahoo.it.invalid>
wrote:

> Hi there,
> I am trying to compile Qpid Proton on Linux Alpine (within a docker image).
> I am getting this error:
> /go/src/qpid-proton/proton-c/src/proactor/epoll.c: In function
> 'pmutex_init':/go/src/qpid-proton/proton-c/src/proactor/epoll.c:104:36:
> error: 'PTHREAD_MUTEX_ADAPTIVE_NP' undeclared (first use in this function)
>  pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_ADAPTIVE_NP);
>                         ^~~~~~~~~~~~~~~~~~~~~~~~~/go/
> src/qpid-proton/proton-c/src/proactor/epoll.c:104:36: note: each
> undeclared identifier is reported only once for each function it appears
> inmake[2]: *** [proton-c/CMakeFiles/qpid-proton.dir/build.make:695:
> proton-c/CMakeFiles/qpid-proton.dir/src/proactor/epoll.c.o] Error
> 1make[1]: *** [CMakeFiles/Makefile2:1069: proton-c/CMakeFiles/qpid-proton.dir/all]
> Error 2make: *** [Makefile:141: all] Error 2
> The relevant part of the Dockerfile (FROM golang:1.8.3-alpine3.6):
> ```# download Apache QPIDWORKDIR /go/src/RUN git clone --progress
> --verbose http://git.apache.org/qpid-proton.git
> # dependencies for Apache QPID (check https://pkgs.alpinelinux.org/
> packages)RUN apk add gcc=6.3.0-r4 g++=6.3.0-r4 cmake=3.8.1-r0
> make=4.2.1-r0RUN apk add python2=2.7.13-r1RUN apk add libssl1.0=1.0.2m-r0
> cyrus-sasl=2.1.26-r10RUN apk add swig=3.0.10-r0
> # compile and install Apache QPIDWORKDIR /go/src/qpid-protonRUN git
> checkout tags/0.18.0WORKDIR /go/src/qpid-proton/buildRUN cmake ..
> -DCMAKE_INSTALL_PREFIX=/usr -DSYSINSTALL_BINDINGS=ONRUN make install```
> The previous steps are based on the details I found here:
> https://github.com/apache/qpid-proton/blob/master/INSTALL.md
> The reason for doing this is to be able to run Go applications using
> Docker and the Electron library for QPID.
> I am not familiar with compiling C projects and I am not able to proceed
> based on that error message.It seems some header files are missing or
> something like that but I can not figure out which dependency is
> missing.Any hint or help on how to compile this library for Linux Alpine?
> Thank youAlessio
>
>


-- 

*Chris Richardson*, System Architect
cr@fourc.eu


*FourC AS, Vestre Rosten 81, Trekanten, NO-7075 Tiller, Norwaywww.fourc.eu
<http://www.fourc.eu/>*

*Follow us on LinkedIn <http://bit.ly/fourcli>, Facebook
<http://bit.ly/fourcfb>, Google+ <http://bit.ly/fourcgp> and Twitter
<http://bit.ly/fourctw>!*