jemalloc 5 incompatibility

["Lausen, Leonard" <la...@amazon.com.INVALID>]

As of jemalloc 5, jemalloc default build can not be used in libraries that are
dlopened. However, libmxnet.so is dlopened by Python (ctypes). To use MXNet with
jemalloc 5, users must not link to system libjemalloc.so but must rather link to
a libjemalloc compiled with special parameters to allow dlopen to work. See 
https://github.com/jemalloc/jemalloc/issues/937

jemalloc 5 is distributed as part of Ubuntu 18.10 and higher, as well as Debian
Stable. Users on these systems will be unable to use MXNet after compiling with
USE_JEMALLOC=ON on systems with libjemalloc-dev, which is the default setting.

Thus in https://github.com/apache/incubator-mxnet/pull/17324 I suggest to
disable Jemalloc by default in the source build. Auto-detecting the version of
jemalloc is not helpful, because over time less and less systems will come with
a working version of jemalloc.

Please go ahead and approve the PR if you agree.

The better solution is to build jemalloc as part of our build. See 
https://github.com/apache/incubator-mxnet/pull/17121
But as CMake build of jemalloc is not yet integrated upstream, this approach
currently relies on a development branch of jemalloc. Thus it was suggested to
revisit the approach once CMake integration is stable upstream.

Re: jemalloc 5 incompatibility

[Marco de Abreu <ma...@gmail.com>]

Building as part of cmake once upstream is stable sounds like a good
approach. Thanks for catching this issue early on!

-Marco

Lausen, Leonard <la...@amazon.com.invalid> schrieb am So., 19. Jan. 2020,
11:20:

> As of jemalloc 5, jemalloc default build can not be used in libraries that
> are
> dlopened. However, libmxnet.so is dlopened by Python (ctypes). To use
> MXNet with
> jemalloc 5, users must not link to system libjemalloc.so but must rather
> link to
> a libjemalloc compiled with special parameters to allow dlopen to work.
> See
> https://github.com/jemalloc/jemalloc/issues/937
>
> jemalloc 5 is distributed as part of Ubuntu 18.10 and higher, as well as
> Debian
> Stable. Users on these systems will be unable to use MXNet after compiling
> with
> USE_JEMALLOC=ON on systems with libjemalloc-dev, which is the default
> setting.
>
> Thus in https://github.com/apache/incubator-mxnet/pull/17324 I suggest to
> disable Jemalloc by default in the source build. Auto-detecting the
> version of
> jemalloc is not helpful, because over time less and less systems will come
> with
> a working version of jemalloc.
>
> Please go ahead and approve the PR if you agree.
>
> The better solution is to build jemalloc as part of our build. See
> https://github.com/apache/incubator-mxnet/pull/17121
> But as CMake build of jemalloc is not yet integrated upstream, this
> approach
> currently relies on a development branch of jemalloc. Thus it was
> suggested to
> revisit the approach once CMake integration is stable upstream.
>