You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "windpiger (via GitHub)" <gi...@apache.org> on 2023/07/23 09:53:50 UTC
[GitHub] [arrow] windpiger opened a new issue, #36827: how to set use_virtual_addressing for pyarrow S3FileSystem
windpiger opened a new issue, #36827:
URL: https://github.com/apache/arrow/issues/36827
### Describe the usage question you have. Please include as many useful details as possible.
Currently pyarrow.fs.S3FileSystem init S3Client by using this logic to set use_virtual_addressing :
https://github.com/apache/arrow/blob/a2a3b9580d66cf40c5df8358f260fdaf5ccc301f/cpp/src/arrow/filesystem/s3fs.cc#L922
https://github.com/apache/arrow/blob/a2a3b9580d66cf40c5df8358f260fdaf5ccc301f/cpp/src/arrow/filesystem/s3fs.cc#L956
But If I want to set endpointOverride to "https://mytest.endpoint" which is not empty, `use_virtual_addressing` will be set to `false` because `use_virtual_addressing = options_.endpoint_override.empty();`
So How can I set `use_virtual_addressing` to `true` if I also want to set a non-empty value for endpointOverride?
Maybe we should de-couple these two configs and set them independently
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] how to set use_virtual_addressing for pyarrow S3FileSystem [arrow]
Posted by "MissiontoMars (via GitHub)" <gi...@apache.org>.
MissiontoMars commented on issue #36827:
URL: https://github.com/apache/arrow/issues/36827#issuecomment-1907305773
This PR only modifies the C++ layer, so when using the Python API, the "force_virtual_addressing" parameter cannot be passed. such as:
`
fs = pyarrow.fs.S3FileSystem(access_key='aaa', secret_key='bbb', endpoint_override='https://xxx', force_virtual_addressing=True)
`
@pitrou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] windpiger commented on issue #36827: how to set use_virtual_addressing for pyarrow S3FileSystem
Posted by "windpiger (via GitHub)" <gi...@apache.org>.
windpiger commented on issue #36827:
URL: https://github.com/apache/arrow/issues/36827#issuecomment-1647209604
@pitrou could you pls help me to review this, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #36827: [C++] how to set use_virtual_addressing for pyarrow S3FileSystem
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #36827:
URL: https://github.com/apache/arrow/issues/36827#issuecomment-1648043184
Also cc @felipecrv if you want an entrypoint into the filesystem subsystem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #36827: [C++] how to set use_virtual_addressing for pyarrow S3FileSystem
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #36827:
URL: https://github.com/apache/arrow/issues/36827#issuecomment-1649850496
> I have another question that how can I use docker-compose to build a distribution wheel on my MacOS?
> `docker-compose python-wheel-manylinux-2014` ? @pitrou
I have no idea. @raulcd Do you know?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] how to set use_virtual_addressing for pyarrow S3FileSystem [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #36827:
URL: https://github.com/apache/arrow/issues/36827#issuecomment-1907671028
Yes, sorry. I should have added a separate issue to expose it in Python.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] windpiger commented on issue #36827: [C++] how to set use_virtual_addressing for pyarrow S3FileSystem
Posted by "windpiger (via GitHub)" <gi...@apache.org>.
windpiger commented on issue #36827:
URL: https://github.com/apache/arrow/issues/36827#issuecomment-1649715863
> Would you like to submit a PR for this?
ok, it's my pleasure~
I have another question that how can I use docker-compose to build a distribution wheel on my MacOS?
`docker-compose python-wheel-manylinux-2014` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #36827: [C++] how to set use_virtual_addressing for pyarrow S3FileSystem
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #36827:
URL: https://github.com/apache/arrow/issues/36827#issuecomment-1647818299
@windpiger When writing this code, I thought that virtual adressing would be incompatible with a custom endpoint, but it turns out the two should work together (the bucket name would be prepended to the custom hostname: `http(s)://<bucket>.endpoint`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] how to set use_virtual_addressing for pyarrow S3FileSystem [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #36827:
URL: https://github.com/apache/arrow/issues/36827#issuecomment-1845578759
This was done in #38858.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on issue #36827: [C++] how to set use_virtual_addressing for pyarrow S3FileSystem
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #36827:
URL: https://github.com/apache/arrow/issues/36827#issuecomment-1727763647
> thank you, guys, I have done with docker-compose python-wheel-manylinux-2014
@windpiger would you mind push to github and create a patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #36827: [C++] how to set use_virtual_addressing for pyarrow S3FileSystem
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #36827:
URL: https://github.com/apache/arrow/issues/36827#issuecomment-1647820948
Would you like to submit a PR for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] how to set use_virtual_addressing for pyarrow S3FileSystem [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #36827: [C++] how to set use_virtual_addressing for pyarrow S3FileSystem
URL: https://github.com/apache/arrow/issues/36827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on issue #36827: [C++] how to set use_virtual_addressing for pyarrow S3FileSystem
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #36827:
URL: https://github.com/apache/arrow/issues/36827#issuecomment-1649859041
I have usually reproduced the python wheel generation on our CI with for example:
```
$ PYTHON=3.11 docker-compose build --no-cache --progress plain python-wheel-manylinux-2014
$ PYTHON=3.11 docker-compose run --rm python-wheel-manylinux-2014
```
You can also run our wheel tests (once generated) with:
```
$ docker-compose build --no-cache --progress plain python-wheel-manylinux-test-unittests
$ docker-compose run --rm python-wheel-manylinux-test-unittests
```
I haven't tested those right now but I have used them in the past.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] windpiger commented on issue #36827: [C++] how to set use_virtual_addressing for pyarrow S3FileSystem
Posted by "windpiger (via GitHub)" <gi...@apache.org>.
windpiger commented on issue #36827:
URL: https://github.com/apache/arrow/issues/36827#issuecomment-1649962994
thank you, guys, I have done with `docker-compose python-wheel-manylinux-2014` @pitrou @raulcd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org