You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datasketches.apache.org by GitBox <gi...@apache.org> on 2023/01/19 14:21:59 UTC
[GitHub] [datasketches-cpp] bllmo opened a new issue, #333: .whl for Linux ARM64 error
bllmo opened a new issue, #333:
URL: https://github.com/apache/datasketches-cpp/issues/333
I have some trouble to install python `datasketches==4.0.0` on Linux ARM64, I receive the following error when I run `pip3 install datasketches==4.0.0`
```
Building wheels for collected packages: datasketches
Building wheel for datasketches (PEP 517) ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 /usr/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpr2gn81le
cwd: /tmp/pip-install-r9ab7pna/datasketches
Complete output (68 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-aarch64-cpython-37
creating build/lib.linux-aarch64-cpython-37/datasketches
copying python/datasketches/PySerDe.py -> build/lib.linux-aarch64-cpython-37/datasketches
copying python/datasketches/__init__.py -> build/lib.linux-aarch64-cpython-37/datasketches
running build_ext
-- The CXX compiler identification is unknown
CMake Error at CMakeLists.txt:25 (project):
No CMAKE_CXX_COMPILER could be found.
Tell CMake where to find the compiler by setting either the environment
variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
to the compiler, or to the compiler name if it is in the PATH.
-- Configuring incomplete, errors occurred!
See also "/tmp/pip-install-r9ab7pna/datasketches/build/temp.linux-aarch64-cpython-37/CMakeFiles/CMakeOutput.log".
See also "/tmp/pip-install-r9ab7pna/datasketches/build/temp.linux-aarch64-cpython-37/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
main()
File "/usr/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/usr/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 205, in build_wheel
metadata_directory)
File "/tmp/pip-build-env-_wmx9kjy/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 414, in build_wheel
wheel_directory, config_settings)
File "/tmp/pip-build-env-_wmx9kjy/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 398, in _build_with_temp_dir
self.run_setup()
File "/tmp/pip-build-env-_wmx9kjy/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 335, in run_setup
exec(code, locals())
File "<string>", line 109, in <module>
File "/tmp/pip-build-env-_wmx9kjy/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 87, in setup
return distutils.core.setup(**attrs)
File "/tmp/pip-build-env-_wmx9kjy/overlay/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
File "/tmp/pip-build-env-_wmx9kjy/overlay/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/tmp/pip-build-env-_wmx9kjy/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/tmp/pip-build-env-_wmx9kjy/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 1208, in run_command
super().run_command(command)
File "/tmp/pip-build-env-_wmx9kjy/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-_wmx9kjy/overlay/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 325, in run
self.run_command("build")
File "/tmp/pip-build-env-_wmx9kjy/overlay/lib/python3.7/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/tmp/pip-build-env-_wmx9kjy/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 1208, in run_command
super().run_command(command)
File "/tmp/pip-build-env-_wmx9kjy/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-_wmx9kjy/overlay/lib/python3.7/site-packages/setuptools/_distutils/command/build.py", line 132, in run
self.run_command(cmd_name)
File "/tmp/pip-build-env-_wmx9kjy/overlay/lib/python3.7/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/tmp/pip-build-env-_wmx9kjy/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 1208, in run_command
super().run_command(command)
File "/tmp/pip-build-env-_wmx9kjy/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "<string>", line 46, in run
File "<string>", line 78, in build_extension
File "/usr/lib64/python3.7/subprocess.py", line 363, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/tmp/pip-install-r9ab7pna/datasketches', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/pip-install-r9ab7pna/datasketches/build/lib.linux-aarch64-cpython-37', '-DWITH_PYTHON=True', '-DCMAKE_CXX_STANDARD=11', '-DPython3_EXECUTABLE=/usr/bin/python3', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
----------------------------------------
ERROR: Failed building wheel for datasketches
Failed to build datasketches
ERROR: Could not build wheels for datasketches which use PEP 517 and cannot be installed directly
```
how can i generate a whl that will work on Linux ARM64 of datasketche==4.0.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-cpp] bllmo commented on issue #333: .whl for Linux ARM64 error
Posted by "bllmo (via GitHub)" <gi...@apache.org>.
bllmo commented on issue #333:
URL: https://github.com/apache/datasketches-cpp/issues/333#issuecomment-1398910221
@jmalkin I can help you test whl directly from my AWS account
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-cpp] jmalkin commented on issue #333: .whl for Linux ARM64 error
Posted by "jmalkin (via GitHub)" <gi...@apache.org>.
jmalkin commented on issue #333:
URL: https://github.com/apache/datasketches-cpp/issues/333#issuecomment-1401124101
Closed this as an issue in that we are now able to generate ARM64 wheels, but it won't be available on pypi until we can run a release cycle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-cpp] jmalkin commented on issue #333: .whl for Linux ARM64 error
Posted by GitBox <gi...@apache.org>.
jmalkin commented on issue #333:
URL: https://github.com/apache/datasketches-cpp/issues/333#issuecomment-1397400142
It looks like we don't have ARM64 on linux as part of the cibuildwheel configuration. I'll fix that going forward.
That said, the error is that it's not finding a compiler. Either g++ or clang++ will work. Does your container include one of those? And if no, can you try an image that does?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-cpp] jmalkin commented on issue #333: .whl for Linux ARM64 error
Posted by "jmalkin (via GitHub)" <gi...@apache.org>.
jmalkin commented on issue #333:
URL: https://github.com/apache/datasketches-cpp/issues/333#issuecomment-1401122737
@will-lauer helped me test this on an AWS instance. Did a quick test of importing and feeding data into a theta sketch, which ensured it both loaded the .so and successfully farmed out calls it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-cpp] jmalkin commented on issue #333: .whl for Linux ARM64 error
Posted by GitBox <gi...@apache.org>.
jmalkin commented on issue #333:
URL: https://github.com/apache/datasketches-cpp/issues/333#issuecomment-1397617578
Ok, the first quick attempts did not work. This might take longer than I'd expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-cpp] jmalkin commented on issue #333: .whl for Linux ARM64 error
Posted by "jmalkin (via GitHub)" <gi...@apache.org>.
jmalkin commented on issue #333:
URL: https://github.com/apache/datasketches-cpp/issues/333#issuecomment-1400709223
@bllmo Please let me know if you get a chance to test it. Hoping we can try for a release and get it into pypi later this week (which will depend on how fast we get votes on the release -- there's a process for Apache releases).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-cpp] bllmo commented on issue #333: .whl for Linux ARM64 error
Posted by GitBox <gi...@apache.org>.
bllmo commented on issue #333:
URL: https://github.com/apache/datasketches-cpp/issues/333#issuecomment-1397447809
thanks for the reply, I need a whl to use on aws glue job that accepts libraries with Python/C only a whl format (aarch64manylinux) [https://docs.aws.amazon.com/glue/latest/dg/author-job-ray-python-libraries.html](see the doc)
> Ray jobs do not support compiling native code in the job environment. This can be a limitation if Python dependencies transiently depend on compiled native code. Ray jobs can run the binaries provided, but they must be compiled for Linux on ARM64. This means that you can use the contents of the aarch64manylinux wheels. You can provide your own native dependencies in compiled form by repackaging a wheel according to Ray standards. Typically, this means removing the dist-info folders so that there is only one folder at the root of the archive.
I use an EC2 with Amazon Linux AMI on arm64.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-cpp] jmalkin commented on issue #333: .whl for Linux ARM64 error
Posted by "jmalkin (via GitHub)" <gi...@apache.org>.
jmalkin commented on issue #333:
URL: https://github.com/apache/datasketches-cpp/issues/333#issuecomment-1410968874
This should now be live in pypi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-cpp] jmalkin commented on issue #333: .whl for Linux ARM64 error
Posted by "jmalkin (via GitHub)" <gi...@apache.org>.
jmalkin commented on issue #333:
URL: https://github.com/apache/datasketches-cpp/issues/333#issuecomment-1398906970
Thanks for that.
I managed to get it "working" last night via emulation. The job took >2hrs to complete, so about 10x slower than the normal wheel generation. But at least it worked, so forward progress.
Trying to test is a challenge currently. I was able to check that the generated .so artifact does appear to be for the right architecture. Interestingly, swapping in that one for the working x86_64 library on my linux desktop produced the same error you got. That makes me suspect there's some sort of mismatch, but also has me a little bit optimistic that the build tool's generic approach might let it auto-select the correct one.
I'm going to make an attempt at cross-compiling since that'd be vastly faster if it works. But since I (might) have a working solution I won't let that be a long blocker in favor of having something to test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-cpp] jmalkin commented on issue #333: .whl for Linux ARM64 error
Posted by GitBox <gi...@apache.org>.
jmalkin commented on issue #333:
URL: https://github.com/apache/datasketches-cpp/issues/333#issuecomment-1397457166
I don't have an immediate fix, unfortunately. I'm running the change to the build workflow, although I don't think I have a way to test it beyond trying to trigger a "wrong architecture" error locally.
If it does seem to work, we can start a release process. There's at least a 3 working day delay baked into that, assuming we can get enough release votes within that window. With luck I'll manage to start that (it'd be 4.0.0.1, I think; this wouldn't even be a patch release) today or tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-cpp] jmalkin commented on issue #333: .whl for Linux ARM64 error
Posted by "jmalkin (via GitHub)" <gi...@apache.org>.
jmalkin commented on issue #333:
URL: https://github.com/apache/datasketches-cpp/issues/333#issuecomment-1398921180
@bllmo It's very much not an official release -- I'm not sure off-hand what changes we've merged since the 4.0.0 release even. But I'm testing in my personal fork just so I don't spam everyone with push and error messages while testing.
The archive with an aarch64 wheel is here: [https://github.com/jmalkin/datasketches-cpp/actions/runs/3964659089](https://github.com/jmalkin/datasketches-cpp/actions/runs/3964659089). There's an "archive.zip" at the bottom. Trying to show my work so you know what you're testing.
It seems to be using python 3.7 based on your previous error messages so I think you'll want datasketches-4.1.20230120.dev425-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-cpp] jmalkin closed issue #333: .whl for Linux ARM64 error
Posted by "jmalkin (via GitHub)" <gi...@apache.org>.
jmalkin closed issue #333: .whl for Linux ARM64 error
URL: https://github.com/apache/datasketches-cpp/issues/333
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-cpp] bllmo commented on issue #333: .whl for Linux ARM64 error
Posted by GitBox <gi...@apache.org>.
bllmo commented on issue #333:
URL: https://github.com/apache/datasketches-cpp/issues/333#issuecomment-1398245925
I did the whl on a AWS AMI linux arm64 machine
`datasketches-4.0.0-cp37-cp37m-linux_aarch64.whl`
but when I run the script on glue I receive this error:
```
{
"Event": "GlueETLJobExceptionEvent",
"Timestamp": 1674213391659,
"Failure Reason": "Traceback (most recent call last):\n File \"/tmp/pip-datasketches.py\", line 8, in <module>\n import datasketches\n File \"/home/spark/.local/lib/python3.7/site-packages/datasketches/__init__.py\", line 20, in <module>\n from .PySerDe import *\n File \"/home/spark/.local/lib/python3.7/site-packages/datasketches/PySerDe.py\", line 18, in <module>\n from _datasketches import PyObjectSerDe\nModuleNotFoundError: No module named '_datasketches'",
"Stack Trace": [
{
"Declaring Class": "<module>",
"Method Name": "from _datasketches import PyObjectSerDe",
"File Name": "/home/spark/.local/lib/python3.7/site-packages/datasketches/PySerDe.py",
"Line Number": 18
},
{
"Declaring Class": "<module>",
"Method Name": "from .PySerDe import *",
"File Name": "/home/spark/.local/lib/python3.7/site-packages/datasketches/__init__.py",
"Line Number": 20
},
{
"Declaring Class": "<module>",
"Method Name": "import datasketches",
"File Name": "/tmp/pip-datasketches.py",
"Line Number": 8
}
],
"Last Executed Line number": 8,
"script": "pip-datasketches.py"
}
```
@jmalkin you might need it for debugging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org