You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "legout (via GitHub)" <gi...@apache.org> on 2023/02/27 13:15:00 UTC
[GitHub] [arrow] legout opened a new issue, #34363: Writing to cloudflare r2 fails for mutlipart upload
legout opened a new issue, #34363:
URL: https://github.com/apache/arrow/issues/34363
### Describe the bug, including details regarding any error messages, version, and platform.
When I try to write a pyarrow.table to cloudflare r2 object store, I got an error when files are larger than x MB (I do not know the exact threshold) and pyarrow internally switches to multipart uploading.
I´ve used `s3fs.S3FileSystem` (fsspec) and also tried `pyarrow.fs.S3FileSystem`. Here is some example code:
```python
import pyarrow.fs as pafs
import s3fs
import pyarrow.parquet as pq
import pyarrow as pa
table_small = pq.read_table("small_data.parquet")
table_large = pq.read_table("large_data.parquet")
fs1 = s3fs.S3FileSystem(
key="some_key",
secret="some_secret",
client_kwargs=dict(endpoint_url="https://123456.r2.cloudflarestorage.com"),
s3_additional_kwargs=dict(ACL="private") # <- this is neccesary for writting.
)
fs2 = pafs.S3FileSystem(
access_key="some_key",
secret_key="some_secret",
endpoint_override="https://123456.r2.cloudflarestorage.com"
)
pq.write_table(table_small, "test/test.parquet", filesystem=fs1) # <- works
pq.write_table(table_small, "test/test.parquet", filesystem=fs2) # <- works
# failed with OSError: [Errno 22] There was a problem with the multipart upload.
pq.write_table(table_large, "test/test.parquet", filesystem=fs1)
# failed with OSError: When initiating multiple part upload for key 'test.parquet' in bucket 'test': AWS Error NETWORK_CONNECTION during CreateMultipartUpload operation: curlCode: 28, Timeout was reached
pq.write_table(table_large, "test/test.parquet", filesystem=fs2)
### Component(s)
Parquet, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-1504210025
Good catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou closed issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
URL: https://github.com/apache/arrow/issues/34363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] elithrar commented on issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "elithrar (via GitHub)" <gi...@apache.org>.
elithrar commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-1574977673
Thank you, @wjones127!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-1500972420
FWIW I am able to reproduce the error by uploading one part of 7MB and one with 6MB, so it does seem like this is the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-1448323248
Thank you for the extra information but I'm not sure it is sufficient. The duckdb issues do not apply. DuckDb has (I think) implemented their own S3 client. Arrow uses [the SDK](https://aws.amazon.com/sdk-for-cpp/). So if there is a bug in URL construction it is an SDK problem and not an Arrow problem. I would be surprised if this were the cause.
The R2 page seems to suggest that CreateMultipartUpload is supported. Also, if this were an R2 compatibility problem then I would expect an "invalid request" error. The error you are getting is "timeout was reached" which seems more likely to be caused by a poor internet connection, a firewall, or a misconfiguration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] legout commented on issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "legout (via GitHub)" <gi...@apache.org>.
legout commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-1448435389
I think it is somehow related to R2, because I am able to run this script (https://github.com/apache/arrow/issues/34363#issue-1601167816) using other s3 object stores (tested with wasabi, contabo s3, idrive e2, storj, aws s3 and self hosted minio) without any problems.
One more information. I am even able to upload the `large_data.parquet` using `s3fs` with `fs1.put_file("large_data.parquet", "test/test.parquet")` and it is also possible with the AWS cli (also uses [the SDK](https://aws.amazon.com/sdk-for-cpp/) ??).
I do understand, that this issue can not be solved within arrow, therefore, we can probably close this here.
However, I´d like to find out what causes this error. Is it possible to run pyarrow commands in a "debugging mode" to get more details?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-1566257562
I'm looking into this now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Writing to cloudflare r2 fails for mutlipart upload [arrow]
Posted by "BitPhinix (via GitHub)" <gi...@apache.org>.
BitPhinix commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-2323762993
Still happening on latest it seems like
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] elithrar commented on issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "elithrar (via GitHub)" <gi...@apache.org>.
elithrar commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-1544274719
(For context: I work at Cloudflare)
Here's the latest:
- R2 _currently_ requires that: all parts are of equal size, a maximum of 10K parts, and per-part limit of 5GB, and a total file size of 5TB.
- DuckDB updated their custom http-fs implementation to work with this
- We don't yet have any concrete plans to relax this in the medium term: s3fs would need to support the ability (or a client using it) to work within R2's requirements
This doesn't mean we're not open to relaxing this, but it's a non-trivial change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] legout commented on issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "legout (via GitHub)" <gi...@apache.org>.
legout commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-1461426688
Here some new information from a Cloudflare developer.
https://github.com/duckdb/duckdb/issues/5685
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] legout commented on issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "legout (via GitHub)" <gi...@apache.org>.
legout commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-1448724033
I run this script (https://github.com/apache/arrow/issues/34363#issue-1601167816) again and it failed with another `OsError`
`OSError: When completing multiple part upload for key 'test2.parquet' in bucket 'test': AWS Error UNKNOWN (HTTP status 400) during CompleteMultipartUpload operation: Unable to parse ExceptionName: InvalidPart Message: There was a problem with the multipart upload.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-1447290499
`pyarrow.fs.S3FileSystem` will always use multi-part upload. However, it seems that multi-part upload is timing out? It seems odd that `CreateMultipartUpload` would fail. This request should be trivial as it is simply setting up the upload and doesn't involve any transmission of data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-1527831607
It seems like we can either
1. Wait for Cloudflare to update R2 to have the same behavior as S3 and Minio
2. Add a configuration option to set the multi-part uploads to always use exactly equal part sizes.
It's possible (2) might be rather invasive, so I'd rather wait a bit to see if (1) happens. But if someone is really motivated to get R2 working now, I think we would accept a PR implementing (2) now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-1500972227
@elithrar in https://github.com/duckdb/duckdb/issues/5685#issuecomment-1456641554 you said
> If the parts aren't equal (minus the last), then the multi-part upload will fail
Should we take that to mean that R2 requires that the parts, except for the last one, must have an equal size in bytes? We treat the part size as a minimum and dynamically increase the size of the parts. This works well for S3.
https://github.com/apache/arrow/blob/28ca876dc41b696bda8159daa1c4e9be2b799c48/cpp/src/arrow/filesystem/s3fs.cc#L1410-L1423
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-1548348207
This seems like a legitimate request and pretty workable. We are pretty close already. The code in ObjectOutputStream is roughly...
```
if request > part_limit:
submit_request(request)
return
buffer.append(request)
if buffer > part_limit:
submit_request(buffer)
buffer.reset()
```
Given we are already talking about cloud upload and I/O I think we can just directly implement the equal parts approach (instead of trying to maintain both) without too much hit to performance (though there will be some hit since this introduces a mandatory extra copy of the data in some cases). This would change the above logic to:
```
buffer.append(request)
for chunk in slice_off_whole_chunks(buffer, part_limit):
submit_request(chunk)
```
Does anyone want to create a PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] legout commented on issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "legout (via GitHub)" <gi...@apache.org>.
legout commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-1447719865
Ok.
Maybe this might be helpful.
https://github.com/duckdb/duckdb/pull/6439
https://github.com/duckdb/duckdb/issues/5685
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Writing to cloudflare r2 fails for mutlipart upload [arrow]
Posted by "geekodour (via GitHub)" <gi...@apache.org>.
geekodour commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-2403553473
@elithrar any updates on if this is going to be implemented for R2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-1544086433
@elithrar Sorry for the ping, but would you have the time to answer @wjones127 's question [here](https://github.com/apache/arrow/issues/34363#issuecomment-1500972227)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] legout commented on issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "legout (via GitHub)" <gi...@apache.org>.
legout commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-1519055485
> FWIW I am able to reproduce the error by uploading one part of 7MB and one with 6MB, so it does seem like this is the issue.
I am able to upload files below 10MB.
For example, I was able to a file with 9.64MB, but failed for a 10.14MB file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] legout commented on issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "legout (via GitHub)" <gi...@apache.org>.
legout commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-1519055411
> Support s3 object store without dynamodb lock delta-io/delta-rs#974
I am able to upload files below 10MB.
For example, I was able to a file with 9.64MB, but failed for a 10.14MB file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #34363: [Python] Writing to cloudflare r2 fails for mutlipart upload
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #34363:
URL: https://github.com/apache/arrow/issues/34363#issuecomment-1452716579
> I´d like to find out what causes this error. Is it possible to run pyarrow commands in a "debugging mode" to get more details?
Try running this before you do anything (before you import `pyarrow.fs`):
```
import pyarrow._s3fs
pyarrow._s3fs.initialize_s3(pyarrow._s3fs.S3LogLevel.Trace)
```
You can also try log levels `Debug`, `Info`, `Warn`. I think it logs to stdout or stderr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org