[jira] [Commented] (ARROW-10921) `TypeError: 'coroutine' object is not iterable` when reading parquet partitions via s3fs >= 0.5 with pyarrow

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249778#comment-17249778 ] 

Joris Van den Bossche commented on ARROW-10921:
-----------------------------------------------

[~inecas] thanks for the report! It's good to have an issue with this error, so others can find it. But so as you already noted yourself (on the PR), this is a duplicate of ARROW-10433 and fixed on master (to be released in pyarrow 3.0.0 in January). For now the best solution is indeed to pin s3fs to <0.5).

> `TypeError: 'coroutine' object is not iterable` when reading parquet partitions via s3fs >= 0.5 with pyarrow
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: ARROW-10921
>                 URL: https://issues.apache.org/jira/browse/ARROW-10921
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>            Reporter: Ivan Necas
>            Priority: Major
>
> Trying to read partitioned parquet files using updated s3fs {{>= 0.5 (using via }}{{dask}}), and got this error:
>  
> {code:python}
> 410 files = set()
>  411 
> --> 412 for key in list(self.fs._ls(path, refresh=refresh)):
>  413 path = key['Key']
>  414 if key['StorageClass'] == 'DIRECTORY':
> TypeError: 'coroutine' object is not iterable
> {code}
> coming from [https://github.com/apache/arrow/blob/9baa123ea38ee9cc1d3a90cfc9347239cd28064c/python/pyarrow/filesystem.py#L415] 
>  
> Seems related to switching s3fs to asyncio in [https://github.com/dask/s3fs/pull/336.|https://github.com/dask/s3fs/pull/336]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)