You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by GitBox <gi...@apache.org> on 2022/12/07 15:32:00 UTC
[GitHub] [arrow] jandom opened a new issue, #14871: pq.ParquetDataset usage with moto3 mocks?
jandom opened a new issue, #14871:
URL: https://github.com/apache/arrow/issues/14871
### Describe the usage question you have. Please include as many useful details as possible.
Hi there,
I'm trying to mock some S3 objects, to write a test exercising a pd.Dataset, this is using boto3, moto3 and pytest
```
@mock_s3
def test_ignore_moto3():
s3 = boto3.resource("s3", region_name="us-east-1")
s3.create_bucket(Bucket="fake-bucket")
parquet_object = s3.Object("fake-bucket", "dummy.parquet")
buffer = io.BytesIO()
df = pd.DataFrame([{"foo": 123, "bar": 123}])
df.to_parquet(buffer, index=False)
parquet_object.put(Body=buffer.getvalue())
s3 = boto3.resource('s3')
obj = s3.Object('fake-bucket', 'dummy.parquet')
print(obj.get()['Body'].read())
ds = pq.ParquetDataset("s3://fake-bucket/dummy.parquet", use_legacy_dataset=False)
```
But unexpectedly this tests is dying
```
> ds = pq.ParquetDataset("s3://fake-bucket/dummy.parquet", use_legacy_dataset=False)
tests/virtual_screening/integration/test_results.py:46:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/opt/micromamba/envs/main/lib/python3.9/site-packages/pyarrow/parquet/core.py:1724: in __new__
return _ParquetDatasetV2(
/opt/micromamba/envs/main/lib/python3.9/site-packages/pyarrow/parquet/core.py:2401: in __init__
if filesystem.get_file_info(path_or_paths).is_file:
pyarrow/_fs.pyx:564: in pyarrow._fs.FileSystem.get_file_info
???
pyarrow/error.pxi:144: in pyarrow.lib.pyarrow_internal_check_status
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E OSError: When getting information for key 'dummy.parquet' in bucket 'fake-bucket': AWS Error ACCESS_DENIED during HeadObject operation: No response body.
```
But the test output includes the file contents (so there is no typo or misconfiguration of the mock)
```
---------------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------------
b'PAR1\x15\x04\x15\x10\x15\x14L\x15\x02\x15\x00\x12\x00\x00\x08\x1c{\x00\x00\x00\x00\x00\x00\x00\x15\x00\x15\x12\x15\x16,\x15\x02\x15\x10\x15\x06\x15\x06\x1c\x18\x08{\x00\x00\x00\x00\x00\x00\x00\x18\x08{\x00\x00\x00\x00\x00\x00\x00\x16\x00(\x08{\x00\x00\x00\x00\x00\x00\x00\x18\x08{\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\t \x02\x00\x00\x00\x02\x01\x01\x02\x00&\xc8\x01\x1c\x15\x04\x195\x10\x00\x06\x19\x18\x03foo\x15\x02\x16\x02\x16\xb8\x01\x16\xc0\x01&8&\x08\x1c\x18\x08{\x00\x00\x00\x00\x00\x00\x00\x18\x08{\x00\x00\x00\x00\x00\x00\x00\x16\x00(\x08{\x00\x00\x00\x00\x00\x00\x00\x18\x08{\x00\x00\x00\x00\x00\x00\x00\x00\x19,\x15\x04\x15\x00\x15\x02\x00\x15\x00\x15\x10\x15\x02\x00\x00\x00\x15\x04\x15\x10\x15\x14L\x15\x02\x15\x00\x12\x00\x00\x08\x1c{\x00\x00\x00\x00\x00\x00\x00\x15\x00\x15\x12\x15\x16,\x15\x02\x15\x10\x15\x06\x15\x06\x1c\x18\x08{\x00\x00\x00\x00\x00\x00\x00\x18\x08{\x00\x00\x00\x00\x00\x00\x00\x16\x00(\x08{\x00\x00\x00\x00\x00\x00\x00\x18\x08{\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x00\t \x02\x00\x00\x00\x02\x01\x01\x02\x00&\xc2\x04\x1c\x15\x04\x195\x10\x00\x06\x19\x18\x03bar\x15\x02\x16\x02\x16\xb8\x01\x16\xc0\x01&\xb2\x03&\x82\x03\x1c\x18\x08{\x00\x00\x00\x00\x00\x00\x00\x18\x08{\x00\x00\x00\x00\x00\x00\x00\x16\x00(\x08{\x00\x00\x00\x00\x00\x00\x00\x18\x08{\x00\x00\x00\x00\x00\x00\x00\x00\x19,\x15\x04\x15\x00\x15\x02\x00\x15\x00\x15\x10\x15\x02\x00\x00\x00\x15\x04\x19<5\x00\x18\x06schema\x15\x04\x00\x15\x04%\x02\x18\x03foo\x00\x15\x04%\x02\x18\x03bar\x00\x16\x02\x19\x1c\x19,&\xc8\x01\x1c\x15\x04\x195\x10\x00\x06\x19\x18\x03foo\x15\x02\x16\x02\x16\xb8\x01\x16\xc0\x01&8&\x08\x1c\x18\x08{\x00\x00\x00\x00\x00\x00\x00\x18\x08{\x00\x00\x00\x00\x00\x00\x00\x16\x00(\x08{\x00\x00\x00\x00\x00\x00\x00\x18\x08{\x00\x00\x00\x00\x00\x00\x00\x00\x19,\x15\x04\x15\x00\x15\x02\x00\x15\x00\x15\x10\x15\x02\x00\x00\x00&\xc2\x04\x1c\x15\x04\x195\x10\x00\x06\x19\x18\x03bar\x15\x02\x16\x02\x16\xb8\x01\x16\xc0\x01&\xb2\x03&\x82\x03\x1c\x18\x08{\x00\x00\x00\x00\x00\x00\x00\x
18\x08{\x00\x00\x00\x00\x00\x00\x00\x16\x00(\x08{\x00\x00\x00\x00\x00\x00\x00\x18\x08{\x00\x00\x00\x00\x00\x00\x00\x00\x19,\x15\x04\x15\x00\x15\x02\x00\x15\x00\x15\x10\x15\x02\x00\x00\x00\x16\xf0\x02\x16\x02&\x08\x16\x80\x03\x14\x00\x00\x19,\x18\x06pandas\x18\xd9\x02{"index_columns": [], "column_indexes": [], "columns": [{"name": "foo", "field_name": "foo", "pandas_type": "int64", "numpy_type": "int64", "metadata": null}, {"name": "bar", "field_name": "bar", "pandas_type": "int64", "numpy_type": "int64", "metadata": null}], "creator": {"library": "pyarrow", "version": "10.0.0"}, "pandas_version": "1.5.1"}\x00\x18\x0cARROW:schema\x18\x80\x06/////zgCAAAQAAAAAAAKAA4ABgAFAAgACgAAAAABBAAQAAAAAAAKAAwAAAAEAAgACgAAAJABAAAEAAAAAQAAAAwAAAAIAAwABAAIAAgAAAAIAAAAEAAAAAYAAABwYW5kYXMAAFkBAAB7ImluZGV4X2NvbHVtbnMiOiBbXSwgImNvbHVtbl9pbmRleGVzIjogW10sICJjb2x1bW5zIjogW3sibmFtZSI6ICJmb28iLCAiZmllbGRfbmFtZSI6ICJmb28iLCAicGFuZGFzX3R5cGUiOiAiaW50NjQiLCAibnVtcHlfdHlwZSI6ICJpbnQ2NCIsICJtZXRhZGF0YSI6IG51bGx9L
CB7Im5hbWUiOiAiYmFyIiwgImZpZWxkX25hbWUiOiAiYmFyIiwgInBhbmRhc190eXBlIjogImludDY0IiwgIm51bXB5X3R5cGUiOiAiaW50NjQiLCAibWV0YWRhdGEiOiBudWxsfV0sICJjcmVhdG9yIjogeyJsaWJyYXJ5IjogInB5YXJyb3ciLCAidmVyc2lvbiI6ICIxMC4wLjAifSwgInBhbmRhc192ZXJzaW9uIjogIjEuNS4xIn0AAAACAAAARAAAAAQAAADU////AAABAhAAAAAUAAAABAAAAAAAAAADAAAAYmFyAMT///8AAAABQAAAABAAFAAIAAYABwAMAAAAEAAQAAAAAAABAhAAAAAcAAAABAAAAAAAAAADAAAAZm9vAAgADAAIAAcACAAAAAAAAAFAAAAA\x00\x18 parquet-cpp-arrow version 10.0.0\x19,\x1c\x00\x00\x1c\x00\x00\x00\x98\x05\x00\x00PAR1'
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jandom commented on issue #14871: pq.ParquetDataset usage with moto3 mocks?
Posted by GitBox <gi...@apache.org>.
jandom commented on issue #14871:
URL: https://github.com/apache/arrow/issues/14871#issuecomment-1341148712
I guess we can close then, I don't see what else can be done here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #14871: pq.ParquetDataset usage with moto3 mocks?
Posted by GitBox <gi...@apache.org>.
pitrou commented on issue #14871:
URL: https://github.com/apache/arrow/issues/14871#issuecomment-1341144580
PyArrow S3 supports doesn't use `boto`, so mocking `boto` won't do anything here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #14871: pq.ParquetDataset usage with moto3 mocks?
Posted by GitBox <gi...@apache.org>.
pitrou commented on issue #14871:
URL: https://github.com/apache/arrow/issues/14871#issuecomment-1341160036
Note you can use something like [Minio](https://min.io/) to still use S3, but backed by a temporary directory on your local filesystem (it's what we do for testing PyArrow's S3 support).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jandom closed issue #14871: pq.ParquetDataset usage with moto3 mocks?
Posted by GitBox <gi...@apache.org>.
jandom closed issue #14871: pq.ParquetDataset usage with moto3 mocks?
URL: https://github.com/apache/arrow/issues/14871
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jandom commented on issue #14871: pq.ParquetDataset usage with moto3 mocks?
Posted by GitBox <gi...@apache.org>.
jandom commented on issue #14871:
URL: https://github.com/apache/arrow/issues/14871#issuecomment-1341148275
That makes perfect sense, thank you :( I'm using S3 to mock it because my application code calls out to S3, we really want to exercise this in an integration test. I guess we could refactor the code to exercise it with local filesystem locations but that's not really the point here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #14871: pq.ParquetDataset usage with moto3 mocks?
Posted by GitBox <gi...@apache.org>.
pitrou commented on issue #14871:
URL: https://github.com/apache/arrow/issues/14871#issuecomment-1341145206
Also, why are you using S3 if just to mock it? Can't you simply write your dataset to a local temporary directory?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org