You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "amoeba (via GitHub)" <gi...@apache.org> on 2023/06/15 21:48:55 UTC
[GitHub] [arrow] amoeba opened a new issue, #36115: [C++][Python][R] Add environment variable for controlling S3 debug level
amoeba opened a new issue, #36115:
URL: https://github.com/apache/arrow/issues/36115
### Describe the enhancement requested
The AWS C++ SDK allows configuring log level which is useful when debugging parts of the Arrow codebase that use S3. Arrow C++ allows direct control of log level during S3 initialization while R gives the user no control and PyArrow has a somewhat hard to use helper that allows the user control log level:
```python
>>> import pyarrow._s3fs # <--- Note the unusual import with the _
>>> pyarrow._s3fs.initialize_s3(pyarrow._s3fs.S3LogLevel.Debug)
```
In addition to the odd import statement above, the user has to make sure they call this early, prior to calling other functions that automatically initialize S3 as a side-effect. See https://github.com/apache/arrow/issues/35575 for discussion. R does not yet have a similar helper but in [the issue to add one](https://github.com/apache/arrow/issues/35260) and the linked Pull Request, it became apparent that a helper function might not be the best way to let users control log level.
Instead of having helpers which may be confusing, an environment variable makes a lot of sense as a way to control this instead. There's already been some interest expressed in https://github.com/apache/arrow/issues/35575 for having an environment variable like `ARROW_S3_LOG_LEVEL` which any initialization code would adapted to respect.
I think there are a few sub-tasks here:
- [ ] Add support in C++, Python, and R for `ARROW_S3_LOG_LEVEL`
- [ ] Include docs for `ARROW_S3_LOG_LEVEL` in C++, Python, R
- [ ] Add `ARROW_S3_LOG_LEVEL` to https://arrow.apache.org/docs/cpp/env_vars.html
### Component(s)
C++, Python, R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python][R] Add environment variable for controlling S3 debug level [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #36115:
URL: https://github.com/apache/arrow/issues/36115#issuecomment-1769715919
Support for this was merged in https://github.com/apache/arrow/pull/38267. Closing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on issue #36115: [C++][Python][R] Add environment variable for controlling S3 debug level
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #36115:
URL: https://github.com/apache/arrow/issues/36115#issuecomment-1638344756
Just adding a +1 to this as I have been running into S3 issues, which I've been using the branch in #35260 to help me diagnose, but having this env var would be extremely useful! @amoeba - if you end up too busy to do this, let me know if I can help!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python][R] Add environment variable for controlling S3 debug level [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba closed issue #36115: [C++][Python][R] Add environment variable for controlling S3 debug level
URL: https://github.com/apache/arrow/issues/36115
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] amoeba commented on issue #36115: [C++][Python][R] Add environment variable for controlling S3 debug level
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #36115:
URL: https://github.com/apache/arrow/issues/36115#issuecomment-1617409652
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python][R] Add environment variable for controlling S3 debug level [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #36115:
URL: https://github.com/apache/arrow/issues/36115#issuecomment-1759910639
I've started working on this and will get a PR in soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org