You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "kimotorc (via GitHub)" <gi...@apache.org> on 2023/06/03 00:09:24 UTC
[GitHub] [arrow] kimotorc opened a new issue, #35894: parquet.read_table opens files twice
kimotorc opened a new issue, #35894:
URL: https://github.com/apache/arrow/issues/35894
### Describe the bug, including details regarding any error messages, version, and platform.
It appears that parquet files are opened twice when using parquet.read_table. The first time seems to be to read in the schema, the second to do the actual data read.
This seems to be the case when use_legacy_dataset=False and without specifying a schema argument. It also is the case for datasets and single parquet reads.
My use case is that i'd like to use a custom fsspec filesystem that interfaces to an s3 like API. It handles the remote download of the parquet file and passes to pyarrow a handle of a temporary file downloaded locally. Without cached access, this behavior results in files being downloaded twice.
[Here](https://github.com/apache/arrow/blob/5a55fb4f66cbd316568085129054b5ff1d2eb089/python/pyarrow/parquet/core.py#L2479) is where the file is opened to read in table schema.
Here is code to reproduce:
```
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
from fsspec.implementations.local import LocalFileSystem
import os
# create a fsspec filesystem to track the number of times files are opened
class LocalFSWithAccessCounter(LocalFileSystem):
def __init__(self, *args, **storage_options):
super().__init__(*args, **storage_options)
self.access_count = {}
def open(
self,
path,
mode="rb",
block_size=None,
cache_options=None,
compression=None,
**kwargs,
):
if path in self.access_count:
self.access_count[path] += 1
else:
self.access_count[path] = 1
return super().open(
path,
mode,
block_size,
cache_options,
compression,
**kwargs,
)
# create a simple test table
df = pd.DataFrame({'col1':range(2),'col2':range(2)})
out_table = pa.Table.from_pandas(df)
# write then read as single parquet file.
# file is opened twice using read_table
pq.write_table(out_table, 'example.parquet')
fs = LocalFSWithAccessCounter()
in_table = pq.read_table('example.parquet', filesystem=fs)
print('files open count:\n', fs.access_count, '\n\n')
#files open count:
# {'example.parquet': 2}
# write then read as dataset with partitions.
# one file is opened twice when using read_table
pq.write_to_dataset(out_table, 'example_ds.parquet', partition_cols=['col1'])
fs = LocalFSWithAccessCounter()
in_table = pq.read_table(os.path.join(os.getcwd(), 'example_ds.parquet'), filesystem=fs)
print('dataset file open count:\n',fs.access_count)
#dataset file open count:
# {'example.parquet': 2, 'example_ds.parquet/col1=0/237b18616a4844b3b094c264f6a09c48-0.parquet': 2, example_ds.parquet/col1=1/2d70ed145ee742e99f4c5528b053446d-0.parquet': 1, }
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #35894: parquet.read_table opens files twice
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #35894:
URL: https://github.com/apache/arrow/issues/35894#issuecomment-1585115025
Do you know which version of pyarrow you were using? There was a fix put in place for 12.0.0 (https://github.com/apache/arrow/issues/33972) that may have addressed this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #35894: parquet.read_table opens files twice
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #35894:
URL: https://github.com/apache/arrow/issues/35894#issuecomment-1599178615
> it seems that was an issue within the Dataset API
Yes, however, pretty much all reads are using the dataset API internally now. The paths have been merged for simplicity of maintenance. `read_table` will create a dataset with one file and then read it. The first time the file is opened happens when the dataset is created (to get the schema of the file). I am pretty sure it isn't actually reading the schema twice though. I think it is something like...
* Create dataset
* open file
* read metadata
* close file
* Read dataset
* open file
* read data
* close file
I agree that it is somewhat less than ideal that the file is opened twice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kimotorc commented on issue #35894: parquet.read_table opens files twice
Posted by "kimotorc (via GitHub)" <gi...@apache.org>.
kimotorc commented on issue #35894:
URL: https://github.com/apache/arrow/issues/35894#issuecomment-1585216512
I am using 12.0.0. I read through [#33972](https://github.com/apache/arrow/issues/33972) and it seems that was an issue within the Dataset API, whereas the extra file opening seems to be initiated from the python shim.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org