You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Alejandro Marco Ramos (Jira)" <ji...@apache.org> on 2022/07/13 10:12:00 UTC
[jira] [Updated] (ARROW-17064) Python hangs when use pyarrow.fs.copy_files is used with "used_threads=True"
[ https://issues.apache.org/jira/browse/ARROW-17064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Marco Ramos updated ARROW-17064:
------------------------------------------
Description:
When try to copy a local path to s3 remote filesystem using `pyarrow.fs.copy_files` and using default parameter `use_threads=True`, the system hangs. If use "use_threads=False` the operation must complete ok (but more slow).
My code is:
{code:java}
>>> import pyarrow as pa
>>> s3fs=pa.fs.S3FileSystem(endpoint_override="http://xxxxxx")
>>> pa.fs.copy_files("tests/data/payments", "bucket/payments", destination_filesystem=s3fs)
... (don't return){code}
If check remote s3, all files appear, but the function don't return
Platform: Windows
was:
When try to copy a local path to s3 remote filesystem using `pyarrow.fs.copy_files` and using default parameter `use_threads=True`, the system hangs. If use "use_threads=False` the operation must complete ok (but more slow).
My code is:
{code:java}
>>> import pyarrow as pa
>>> s3fs=pa.fs.S3FileSystem(endpoint_override="http://xxxxxx")
>>> pa.fs.copy_files("tests/data/payments", "bucket/payments", destination_filesystem=s3fs)
... (don't return){code}
If check remote s3, all files appear, but the function don't return
> Python hangs when use pyarrow.fs.copy_files is used with "used_threads=True"
> ----------------------------------------------------------------------------
>
> Key: ARROW-17064
> URL: https://issues.apache.org/jira/browse/ARROW-17064
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 8.0.0
> Reporter: Alejandro Marco Ramos
> Priority: Major
>
> When try to copy a local path to s3 remote filesystem using `pyarrow.fs.copy_files` and using default parameter `use_threads=True`, the system hangs. If use "use_threads=False` the operation must complete ok (but more slow).
>
> My code is:
> {code:java}
> >>> import pyarrow as pa
> >>> s3fs=pa.fs.S3FileSystem(endpoint_override="http://xxxxxx")
> >>> pa.fs.copy_files("tests/data/payments", "bucket/payments", destination_filesystem=s3fs)
> ... (don't return){code}
> If check remote s3, all files appear, but the function don't return
>
> Platform: Windows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)