You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pylucene-dev@lucene.apache.org by "Wang, Clem" <cw...@conductiv.com> on 2021/02/10 03:11:54 UTC
Trouble with building PyLucene on Mac
(My msg originally post here: https://issues.apache.org/jira/projects/PYLUCENE/issues/PYLUCENE-10 but Andreas Vajda said I should send to the mailing list. I missed whatever he had posted to the mailing list previously )
I had a lot of trouble with building PyLucene. I finally got it built (under Python 2.7) with much gnashing of teeth but I'm not sure I can get it to run under 2.7 (I'm not going to bother describing that since I don't think it will be useful.) I'd rather try to get PyLucene built and running under Python 3.x.
I suspect some of this has to do with lack of support from Apple and Oracle for Java as well as half-hearted support between gcc & clang headers, libraries, and flags, so I'm not even sure how much responsibility falls onto PyLucene.
For me, it would be preferable if I could just download a binary (although I don't know how difficult that would be).
My configuration:
* pylucene-8.6.1 gotten from https://mirrors.ocf.berkeley.edu/apache/lucene/pylucene/
* Mac OSX 10.15.7
* Macbook Pro, Intel Core i7
* gcc --version gcc (Homebrew GCC 10.2.0_3) 10.2.0
* clang --version
* Apple clang version 12.0.0 (clang-1200.0.32.29)
* Target: x86_64-apple-darwin19.6.0
* Thread model: posix
* Python 3.8.2
* java -version
* openjdk version "1.8.0_282"
* OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_282-b08)
* OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.282-b08, mixed mode)
I get 4 errors I get when building with Python 3.8.2, of which one is easy to fix but the other are total puzzles to me:
python setup.py build
...
/opt/local/bin/gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -I/usr/local/opt/libomp/include -Xpreprocessor -fopenmp -dynamiclib -D_jcc_lib -DJCC_VER="3.8" -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I_jcc3 -Ijcc3/sources -I/Users/cwang/3.7/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c jcc3/sources/jcc.cpp -o build/temp.macosx-10.14.6-x86_64-3.8/jcc3/sources/jcc.o -DPYTHON -fno-strict-aliasing -Wno-write-strings -mmacosx-version-min=10.9 -std=c++11 -stdlib=libc++
gcc: error: this compiler does not support arm64
gcc: error: unrecognized command-line option '-iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders'
gcc: error: unrecognized command-line option '-iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers'
gcc: error: unrecognized command-line option '-stdlib=libc++'
Error 1. this makes least amount of sense is having two different architecture flags when I'm building on and for an Intel machine. However, I have no idea where this is being mandated. (I can't find it in the setup.py file)
flags -arch arm64 -arch x86_64
Error 2 & 3 couldn't find this string in setup.py and nothing actionable with google:
-iwithsysroot/....
but it does appear to be a clang flag: https://clang.llvm.org/docs/ClangCommandLineReference.html
Error 4 seems easy to fix:
-stdlib=libc++
it looks like shouldn't be used and that was easy to remove from setup.py:
https://stackoverflow.com/questions/19774778/when-is-it-necessary-to-use-the-flag-stdlib-libstdc
I also believe that
-ljvm
and
-ljava
flags need to be removed as well from the LFLAGS for darwin.
This message and its contents are PROPRIETARY AND CONFIDENTIAL to Premier, Inc. inclusive of its affiliates and subsidiaries. This message and all attachments are a private communication and are only for the intended recipient. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the information contained in or attached to this message is strictly prohibited. Please notify the sender of the delivery error by replying to this message, and then delete it from your system.
Re: Trouble with building PyLucene on Mac
Posted by Andi Vajda <va...@apache.org>.
Hi Clem,
Lots of replies inline...
On Wed, 10 Feb 2021, Wang, Clem wrote:
> (My msg originally post here:
> https://issues.apache.org/jira/projects/PYLUCENE/issues/PYLUCENE-10 but
> Andreas Vajda said I should send to the mailing list. I missed whatever
> he had posted to the mailing list previously )
For earlier posts, you may refer to the mailing list archives:
https://mail-archives.apache.org/mod_mbox/lucene-pylucene-dev/
> I had a lot of trouble with building PyLucene. I finally got it built
> (under Python 2.7) with much gnashing of teeth but I'm not sure I can get
> it to run under 2.7 (I'm not going to bother describing that since I don't
> think it will be useful.) I'd rather try to get PyLucene built and running
> under Python 3.x.
Please, don't complain or gnash your teeth, PyLucene is a C++/Java/Python
extension and knowledge on how to operate a C++ compiler and linker is
required. No excuses. The PyLucene build uses a Makefile with example
configurations that you need to modify to satisfy your environment's
constraints. Before building PyLucene, you need to build JCC, which uses a
setup.py file where, again, depending on your setup (Java, in particular),
you need to set things that fit your installation.
> I suspect some of this has to do with lack of support from Apple and
> Oracle for Java as well as half-hearted support between gcc & clang
> headers, libraries, and flags, so I'm not even sure how much
> responsibility falls onto PyLucene.
No amount of blaming others is going to spare you the need to know
how to operate a C++ compiler and linker competently.
Last I checked, Apple didn't include a version of Python 3 in their OS, so,
if you wish to run PyLucene with Python 3, you need to start by installing
Python 3, either from sources or from a binary distribution of your choice.
I suggest starting with sources at python.org.
> For me, it would be preferable if I could just download a binary (although
> I don't know how difficult that would be).
I can't distribute binaries. There are too many combinations of binaries
to build based on all the OS, Python, Java and Lucene versions. I also
refuse to take responsibility in shipping binaries, as I can't vouch for
them. What if they contain a virus ? You can inspect all the sources,
however.
PyLucene is open source and you are expected to be able to build it from
sources without too much trouble and debug issues related to your
environment choices yourself. You are also welcome to ask questions here and
we can help you, but complaining about things or gnashing teeth is going to
waste the goodwill of readers of this list.
> My configuration:
>
> * pylucene-8.6.1 gotten from https://mirrors.ocf.berkeley.edu/apache/lucene/pylucene/
>
> * Mac OSX 10.15.7
> * Macbook Pro, Intel Core i7
> * gcc --version gcc (Homebrew GCC 10.2.0_3) 10.2.0
> * clang --version
> * Apple clang version 12.0.0 (clang-1200.0.32.29)
> * Target: x86_64-apple-darwin19.6.0
> * Thread model: posix
You must use the same compiler that was used to build the Python version of
your choise. If you installed binaries from python.org, that is likely to
be clang from Apple's command line dev tools.
If it's from homebrew, then probably a homebrew compiler. But, by all means,
do not mix homebrew and non-homebrew things.
I recommend you take the homebrew gcc compiler off your PATH and ensure you
have the Apple Clang from the command line dev tools installed and
accessible on your PATH by default (or from XCode) and build Python 3 from
sources. See xcodeselect for more information.
https://developer.apple.com/library/archive/technotes/tn2339/_index.html
These are the steps I used to configure and build Python 3.9.1 a couple of
weeks ago after downloading the sources from python.org:
step 0: build and install libressl from sources
$ curl -O https://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-<latest>.tar.gz
$ tar -xvzf ~/tmp/downloads/libressl-<version>.tar.gz
$ cd libressl-<version>
$ ./configure --prefix=/usr/local/libressl
$ make
$ sudo make install
step 1:
download python 3 sources from python.org
unpack the archive
$ ./configure --prefix=`pwd`/_install --enable-framework=`pwd`/_framework --with-openssl=/usr/local/libressl
$ make
$ make install
(python is now installed in `pwd`/_install, you may of course, choose
to install it anywhere)
I also strongly recommend you setup a virtual environment for PyLucene
at this time.
Now that you have python 3 installed, onto JCC
step 0: ensure you have a Java JDK installed
step 1: ensure JCC's setup.py is properly configured for your setup
$ `pwd`/_install/bin/python setup.py build install
(or use the path to python in your virtual env, if you set one up)
Once JCC built installed, edit PyLucene's Makefile to fit your environment
as well.
$ make
$ make test
$ make install
> * Python 3.8.2
> * java -version
> * openjdk version "1.8.0_282"
> * OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_282-b08)
> * OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.282-b08, mixed mode)
>
> I get 4 errors I get when building with Python 3.8.2, of which one is easy to fix but the other are total puzzles to me:
>
> python setup.py build
>
> ...
>
> /opt/local/bin/gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -I/usr/local/opt/libomp/include -Xpreprocessor -fopenmp -dynamiclib -D_jcc_lib -DJCC_VER="3.8" -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I_jcc3 -Ijcc3/sources -I/Users/cwang/3.7/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c jcc3/sources/jcc.cpp -o build/temp.macosx-10.14.6-x86_64-3.8/jcc3/sources/jcc.o -DPYTHON -fno-strict-aliasing -Wno-write-strings -mmacosx-version-min=10.9 -std=c++11 -stdlib=libc++
>
> gcc: error: this compiler does not support arm64
Do not mix homebrew with non-homebrew.
Python, by way of setup.py, will pass flags to the C++ compilers that it was
built with. If python was built for arm64 as well, then this is going to
happen unless override the default flags. It's better to use the same compiler
that was used to build python.
> gcc: error: unrecognized command-line option '-iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders'
>
> gcc: error: unrecognized command-line option '-iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers'
>
> gcc: error: unrecognized command-line option '-stdlib=libc++'
These are issues of homebrew vs non-homebrew.
If Python was built with Apple's Clang, use Apple's clang for everything else.
> Error 1. this makes least amount of sense is having two different architecture flags when I'm building on and for an Intel machine. However, I have no idea where this is being mandated. (I can't find it in the setup.py file)
>
> flags -arch arm64 -arch x86_64
By default, Apple likes to build for all architectures it supports. The Apple
M1 chip corresponds to 'arm64'. See above how to work this around.
> Error 2 & 3 couldn't find this string in setup.py and nothing actionable with google:
>
> -iwithsysroot/....
>
> but it does appear to be a clang flag:
>https://clang.llvm.org/docs/ClangCommandLineReference.html
You are not using clang, but homebrew's gcc, apparently.
>
> Error 4 seems easy to fix:
>
> -stdlib=libc++
>
> it looks like shouldn't be used and that was easy to remove from setup.py:
>
> https://stackoverflow.com/questions/19774778/when-is-it-necessary-to-use-the-flag-stdlib-libstdc
>
>
>
> I also believe that
>
> -ljvm
>
> and
>
> -ljava
>
> flags need to be removed as well from the LFLAGS for darwin.
That could very well be the case with the JDK you picked.
I'm building, testing and developing with adoptopenjdk, not Oracle's.
You need to edit setup.py to match your environment choices.
> This message and its contents are PROPRIETARY AND CONFIDENTIAL to Premier, Inc. inclusive of its affiliates and subsidiaries. This message and all attachments are a private communication and are only for the intended recipient. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the information contained in or attached to this message is strictly prohibited. Please notify the sender of the delivery error by replying to this message, and then delete it from your system.
No, this message doesn't contain anything proprietary.
If you have more questions, please don't hesitate to ask but no complaints ;-)
Andi..