You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "shou123 (via GitHub)" <gi...@apache.org> on 2023/04/27 20:00:51 UTC
[GitHub] [arrow] shou123 opened a new issue, #35357: [python] [Skyhook data management issue] the code not include the "libarrow_skyhook_client.so" library
shou123 opened a new issue, #35357:
URL: https://github.com/apache/arrow/issues/35357
### Describe the bug, including details regarding any error messages, version, and platform.
According to the paper mention:
import pyarrow.dataset as ds
format_ = ds.SkyhookFileFormat( "parquet", "/ceph.conf" )
But for the skyhook build, when set up "ARROW_SKYHOOK=ON", there is no "libarrow_skyhook_client.so" library generate. SkyhookFileFormat API can not be used.
### Component(s)
Packaging, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] shou123 commented on issue #35357: [Python][Skyhook] the code not include the "libarrow_skyhook_client.so" library
Posted by "shou123 (via GitHub)" <gi...@apache.org>.
shou123 commented on issue #35357:
URL: https://github.com/apache/arrow/issues/35357#issuecomment-1556424690
> The libarrow_skyhook_client.so library is generated by the arrow_skyhook_client target in the pyarrow/cpp/build/BUILD.gn file. This target is only enabled when ARROW_SKYHOOK=ON is set.
>
> The SkyhookFileFormat API is implemented in the arrow/ipc/skyhook.cc file. This file only includes the libarrow_skyhook_client.so library if it is available.
>
> When ARROW_SKYHOOK=ON is not set, the libarrow_skyhook_client.so library is not generated, and the SkyhookFileFormat API is not available.
PS, I also set the 'ARROW_SKYHOOK=ON' and according to the paper: 'https://arxiv.org/pdf/2204.06074.pdf" paper, the pyarrow need to include a function named 'SkyhookFileFormat'. But it is not include this function at 'pyarrow ' library.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] shou123 commented on issue #35357: [Python][Skyhook] the code not include the "libarrow_skyhook_client.so" library
Posted by "shou123 (via GitHub)" <gi...@apache.org>.
shou123 commented on issue #35357:
URL: https://github.com/apache/arrow/issues/35357#issuecomment-1527552949
> Skyhook's library name is `libarrow_skyhook.so` not `libarrow_skyhook_client.so`. Why do you think that the name is `libarrow_skyhook_client.so`.
>
> Cc: @JayjeetAtGithub
'libarrow_skyhook_client.so' is support to using 'SkyhookFileFormat' API which is link 'arrow_dataset', 'arrow', and 'arrow_skyhook_client' shared libraries during compiling.
Reference: [https://jayjeetc.medium.com/skyhookdm-is-now-a-part-of-apache-arrow-e5d7b9a810ba]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][Skyhook] the code not include the "libarrow_skyhook_client.so" library [arrow]
Posted by "drin (via GitHub)" <gi...@apache.org>.
drin commented on issue #35357:
URL: https://github.com/apache/arrow/issues/35357#issuecomment-2000125070
@shou123 , I know this is quite late, but did you manage to figure out your issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] shou123 commented on issue #35357: [Python][Skyhook] the code not include the "libarrow_skyhook_client.so" library
Posted by "shou123 (via GitHub)" <gi...@apache.org>.
shou123 commented on issue #35357:
URL: https://github.com/apache/arrow/issues/35357#issuecomment-1556421021
> The libarrow_skyhook_client.so library is generated by the arrow_skyhook_client target in the pyarrow/cpp/build/BUILD.gn file. This target is only enabled when ARROW_SKYHOOK=ON is set.
Sorry, I didn't find the "pyarrow/cpp/build/BUILD.gn" file in the aparche arrow source code. Could you please help provide a source code link?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] shou123 commented on issue #35357: [Python][Skyhook] the code not include the "libarrow_skyhook_client.so" library
Posted by "shou123 (via GitHub)" <gi...@apache.org>.
shou123 commented on issue #35357:
URL: https://github.com/apache/arrow/issues/35357#issuecomment-1575686749
> How are you installing Arrow today? I think we might not be enabling skyhook in the wheels that we publish to pypi / conda-forge. So you will have to build wheels from source. Directions on how to do this are here: https://arrow.apache.org/docs/developers/python.html
Thank you for providing the information. I'll try for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Rajneesh2223 commented on issue #35357: [Python][Skyhook] the code not include the "libarrow_skyhook_client.so" library
Posted by "Rajneesh2223 (via GitHub)" <gi...@apache.org>.
Rajneesh2223 commented on issue #35357:
URL: https://github.com/apache/arrow/issues/35357#issuecomment-1555970746
The libarrow_skyhook_client.so library is generated by the arrow_skyhook_client target in the pyarrow/cpp/build/BUILD.gn file. This target is only enabled when ARROW_SKYHOOK=ON is set.
The SkyhookFileFormat API is implemented in the arrow/ipc/skyhook.cc file. This file only includes the libarrow_skyhook_client.so library if it is available.
When ARROW_SKYHOOK=ON is not set, the libarrow_skyhook_client.so library is not generated, and the SkyhookFileFormat API is not available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #35357: [Python][Skyhook] the code not include the "libarrow_skyhook_client.so" library
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #35357:
URL: https://github.com/apache/arrow/issues/35357#issuecomment-1571969248
How are you installing Arrow today? I think we might not be enabling skyhook in the wheels that we publish to pypi / conda-forge. So you will have to build wheels from source. Directions on how to do this are here: https://arrow.apache.org/docs/developers/python.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] shou123 commented on issue #35357: [Python][Skyhook] the code not include the "libarrow_skyhook_client.so" library
Posted by "shou123 (via GitHub)" <gi...@apache.org>.
shou123 commented on issue #35357:
URL: https://github.com/apache/arrow/issues/35357#issuecomment-1529116618
> The article was written by @JayjeetAtGithub . So we should wait for a response from @JayjeetAtGithub . :-)
For Sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #35357: [Python][Skyhook] the code not include the "libarrow_skyhook_client.so" library
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #35357:
URL: https://github.com/apache/arrow/issues/35357#issuecomment-1526834668
Skyhook's library name is `libarrow_skyhook.so` not `libarrow_skyhook_client.so`.
Why do you think that the name is `libarrow_skyhook_client.so`.
Cc: @JayjeetAtGithub
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #35357: [Python][Skyhook] the code not include the "libarrow_skyhook_client.so" library
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #35357:
URL: https://github.com/apache/arrow/issues/35357#issuecomment-1528445287
The article was written by @JayjeetAtGithub . So we should wait for a response from @JayjeetAtGithub . :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org