You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "amoeba (via GitHub)" <gi...@apache.org> on 2023/04/20 23:05:35 UTC
[GitHub] [arrow] amoeba opened a new issue, #35260: [R] Allow users to adjust AWS S3 log level from R
amoeba opened a new issue, #35260:
URL: https://github.com/apache/arrow/issues/35260
### Describe the enhancement requested
The AWS SDK has built-in logging and the C++ Arrow S3 implementation provides a way to initialize S3 with a log level other than the default (Fatal). Setting the log level above the default of Fatal is useful for both users and developers who might want to understand what the AWS SDK is doing behind the scenes. My immediate use case is to measure the timing and number of HTTP requests `arrow::write_dataset` triggers for a workload I'm troubleshooting.
The Python bindings expose such an option as:
```python
import pyarrow
from pyarrow import fs
fs.initialize_s3(fs.S3LogLevel.Debug)
```
As far as I can tell, the R bindings don't expose anything similar. Would this be worth a PR? It could look something like:
```r
library(arrow)
arrow:::s3_init("Debug") # Output starts printing to the console
s3_bucket(....) # Prints lots of debug info to console
```
I'm not sure whether my example `s3_init` should be exported or not so I'm also curious for thoughts on that.
### Component(s)
R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] amoeba commented on issue #35260: [R] Allow users to adjust AWS S3 log level from R
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #35260:
URL: https://github.com/apache/arrow/issues/35260#issuecomment-1532361346
Hey @thisisnic, I've sent off a PR in https://github.com/apache/arrow/pull/35398. Let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on issue #35260: [R] Allow users to adjust AWS S3 log level from R
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #35260:
URL: https://github.com/apache/arrow/issues/35260#issuecomment-1523289370
Anything which makes debugging easier for people who are using the package is definitely worth a PR!
> I'm not sure whether my example s3_init should be exported or not so I'm also curious for thoughts on that.
Sounds reasonable enough to me right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R] Allow users to adjust AWS S3 log level from R [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou closed issue #35260: [R] Allow users to adjust AWS S3 log level from R
URL: https://github.com/apache/arrow/issues/35260
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on issue #35260: [R] Allow users to adjust AWS S3 log level from R
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #35260:
URL: https://github.com/apache/arrow/issues/35260#issuecomment-1594270392
Thanks @amoeba. I think that an environment variable is a great idea, and I appreciate you having the patience to continue to engage with this and enduring all of the "this solution doesn't feel quite right, but I don't have a good alternative to suggest to you" feedback that it took to get here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] amoeba commented on issue #35260: [R] Allow users to adjust AWS S3 log level from R
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #35260:
URL: https://github.com/apache/arrow/issues/35260#issuecomment-1526758242
Thanks @thisisnic! I'll get a PR in soon and give you a ping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] amoeba commented on issue #35260: [R] Allow users to adjust AWS S3 log level from R
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #35260:
URL: https://github.com/apache/arrow/issues/35260#issuecomment-1593765576
Following from discussion in https://github.com/apache/arrow/issues/35575, I've filed https://github.com/apache/arrow/issues/36115 which would add an environment variable, `ARROW_S3_LOG_LEVEL` which might be an easier route to allow control of the S3 log level than this issue. If such an environment variable existed, I'm not we'd need the change described in this issue, depending on how https://github.com/apache/arrow/issues/36115 goes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org