You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "davlee1972 (via GitHub)" <gi...@apache.org> on 2023/06/21 21:09:07 UTC
[GitHub] [arrow] davlee1972 opened a new issue, #36223: Windows S3FileSystem - AWS Error NETWORK_CONNECTION during HeadObject operation
davlee1972 opened a new issue, #36223:
URL: https://github.com/apache/arrow/issues/36223
### Describe the bug, including details regarding any error messages, version, and platform.
I can't get S3FileSystem to work on Windows.. The same code works fine on Linux..
The AWS boto3 python library works fine on windows, but pyarrow is returning the following error:
```
>>> s3.get_file_info("my_bucket/model-portfolios.json")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyarrow\_fs.pyx", line 571, in pyarrow._fs.FileSystem.get_file_info
File "pyarrow\error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow\error.pxi", line 115, in pyarrow.lib.check_status
OSError: When getting information for key 'model-portfolios.json' in bucket 'my_bucket': AWS Error NETWORK_CONNECTION during HeadObject operation: Encountered network error when sending http request
```
On linux this runs just fine.
```
>>> s3.get_file_info("my_bucket/model-portfolios.json")
<FileInfo for 'my_bucket/model-portfolios.json': type=FileType.File, size=138144>
```
boto3 on the same windows machine works fine.
```
>>> s3_config = Config(connect_timeout=30, retries={"max_attempts": 1})
>>> s3 = boto3.client(
... "s3",
... aws_access_key_id="....",
... aws_secret_access_key="....",
... endpoint_url="....",
... verify=True,
... config=s3_config,
... )
>>>
>>> resp_obj = s3.get_object(
... Bucket="my_bucket",
... Key="model-portfolios.json"
... )
>>>
>>> resp_obj
{'ResponseMetadata': {'RequestId': '1687381629753325', 'HostId': '12527701', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Wed, 21 Jun 2023 21:07:09 GMT', 'connection': 'KEEP-ALIVE', 'server': 'StorageGRID/11.6.0.7', 'x-amz-request-id': '1687381629753325', 'x-amz-id-2': '12527701', 'content-length': '138144', 'x-ntap-sg-trace-id': '9b8092e31d357aa4', 'etag': '"67356b777d74c255b2ec58b7d1e810f9"', 'x-amz-server-side-encryption': 'AES256', 'content-type': 'application/json', 'last-modified': 'Mon, 01 Jul 2019 21:12:53 GMT', 'accept-ranges': 'bytes'}, 'RetryAttempts': 0}, 'AcceptRanges': 'bytes', 'LastModified': datetime.datetime(2019, 7, 1, 21, 12, 53, tzinfo=tzutc()), 'ContentLength': 138144, 'ETag': '"67356b777d74c255b2ec58b7d1e810f9"', 'ContentType': 'application/json', 'ServerSideEncryption': 'AES256', 'Metadata': {}, 'Body': <botocore.response.StreamingBody object at 0x000001CE17DD0880>}
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] davlee1972 closed issue #36223: Windows S3FileSystem - AWS Error NETWORK_CONNECTION during HeadObject operation
Posted by "davlee1972 (via GitHub)" <gi...@apache.org>.
davlee1972 closed issue #36223: Windows S3FileSystem - AWS Error NETWORK_CONNECTION during HeadObject operation
URL: https://github.com/apache/arrow/issues/36223
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #36223: Windows S3FileSystem - AWS Error NETWORK_CONNECTION during HeadObject operation
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #36223:
URL: https://github.com/apache/arrow/issues/36223#issuecomment-1613982437
Can you add these lines to the top of your script? It will enable trace logging with S3:
```
import pyarrow._s3fs
pyarrow._s3fs.initialize_s3(pyarrow._s3fs.S3LogLevel.Trace)
```
Note: this must be done before any other imports (specifically before `import pyarrow.fs`). You'll know if it succeeds (the log is very noisy).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] davlee1972 commented on issue #36223: Windows S3FileSystem - AWS Error NETWORK_CONNECTION during HeadObject operation
Posted by "davlee1972 (via GitHub)" <gi...@apache.org>.
davlee1972 commented on issue #36223:
URL: https://github.com/apache/arrow/issues/36223#issuecomment-1710866765
This error disappeared on its own. I suspect someone on our S3 server was seeing these errors and made some sort of network configuration change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org