You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@libcloud.apache.org by GitBox <gi...@apache.org> on 2022/03/12 12:17:02 UTC
[GitHub] [libcloud] Germandrummer92 opened a new pull request #1664: Exception Handling for S3 storage requests
Germandrummer92 opened a new pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664
## Exception Handling for S3 storage requests
### Description
We are currently again running into some errors again unfortunately when using libcloud and the storage buckets scaling up/returning 500s under load. However after spending a couple of hours on it: I believe the libcloud implementation of the retrying is correct and should work.
I did add a couple of tests however and improved one small thing: The RawResponse now ALSO parses the error as the base Response class does. At best we would actually call the super init in the RawResponse class, but that is not possible due to the overriding of mutable properties with computed ones, I might look into that some other time.
This means we can remove custom error parsing that was added for example in the storage upload file and can rely on the responses parsing their own errors instead.
Imho this makes the flow clearer since it's the same for both Response types, and removes the "duplicate" error parsing in the case of a non-raw-response.
Feel free to not use this, but I wanted to open a PR anyways since I spent some time on it.
### Status
- done, ready for review
### Checklist (tick everything that applies)
- [X] [Code linting](http://libcloud.readthedocs.org/en/latest/development.html#code-style-guide) (required, can be done after the PR checks)
- [X] Documentation
- [X] [Tests](http://libcloud.readthedocs.org/en/latest/testing.html)
- [X] [ICLA](http://libcloud.readthedocs.org/en/latest/development.html#contributing-bigger-changes) (required for bigger changes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] codecov-commenter commented on pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#issuecomment-1066069396
# [Codecov](https://codecov.io/gh/apache/libcloud/pull/1664?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1664](https://codecov.io/gh/apache/libcloud/pull/1664?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a7a1521) into [trunk](https://codecov.io/gh/apache/libcloud/commit/d72d4a3328a443ead76ed3bc96ac9c83bbd41b06?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d72d4a3) will **increase** coverage by `0.00%`.
> The diff coverage is `76.81%`.
[](https://codecov.io/gh/apache/libcloud/pull/1664?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## trunk #1664 +/- ##
=======================================
Coverage 83.25% 83.25%
=======================================
Files 400 400
Lines 87113 87168 +55
Branches 9249 9252 +3
=======================================
+ Hits 72524 72575 +51
- Misses 11446 11450 +4
Partials 3143 3143
```
| [Impacted Files](https://codecov.io/gh/apache/libcloud/pull/1664?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [libcloud/storage/base.py](https://codecov.io/gh/apache/libcloud/pull/1664/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGliY2xvdWQvc3RvcmFnZS9iYXNlLnB5) | `82.82% <ø> (-0.18%)` | :arrow_down: |
| [libcloud/test/storage/test\_local.py](https://codecov.io/gh/apache/libcloud/pull/1664/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGliY2xvdWQvdGVzdC9zdG9yYWdlL3Rlc3RfbG9jYWwucHk=) | `91.26% <13.33%> (-0.76%)` | :arrow_down: |
| [libcloud/test/test\_connection.py](https://codecov.io/gh/apache/libcloud/pull/1664/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGliY2xvdWQvdGVzdC90ZXN0X2Nvbm5lY3Rpb24ucHk=) | `97.94% <92.50%> (-0.63%)` | :arrow_down: |
| [libcloud/common/base.py](https://codecov.io/gh/apache/libcloud/pull/1664/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGliY2xvdWQvY29tbW9uL2Jhc2UucHk=) | `90.83% <100.00%> (+0.60%)` | :arrow_up: |
| [libcloud/test/storage/test\_base.py](https://codecov.io/gh/apache/libcloud/pull/1664/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGliY2xvdWQvdGVzdC9zdG9yYWdlL3Rlc3RfYmFzZS5weQ==) | `95.95% <100.00%> (+0.30%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/libcloud/pull/1664?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/libcloud/pull/1664?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [d72d4a3...a7a1521](https://codecov.io/gh/apache/libcloud/pull/1664?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] RunOrVeith commented on pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
RunOrVeith commented on pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#issuecomment-1077458944
Sadly installing directly from git kills the dependency resolving of poetry for no apparent reason (and induces a slow down of factor 100x before crashing), so it would be really great to get a patch release

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] RunOrVeith commented on pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
RunOrVeith commented on pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#issuecomment-1077576056
I tried it with the zip and it seems to reduce the resolving time by about 25-30%, however it produced a wrong dependency with the cryptography library that broke the build. So I will just deal with the longer resolving time and some manual hacking the lock file until it is released
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami merged pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
Kami merged pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Germandrummer92 commented on pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
Germandrummer92 commented on pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#issuecomment-1071360638
We managed now, thanks anyways!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Germandrummer92 edited a comment on pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
Germandrummer92 edited a comment on pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#issuecomment-1068931986
sorry to bother you again @Kami: but any chance we can get a patch release with this change? It's proving surprisingly difficult on our side unfortunately to introduce a git dependency due to some poetry intricacies it turns out, an official release would make this much easier
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Germandrummer92 commented on pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
Germandrummer92 commented on pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#issuecomment-1067672516
this one good to go @Kami? Would be great for us to get it out quickly 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on a change in pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
Kami commented on a change in pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#discussion_r825423677
##########
File path: libcloud/test/storage/test_local.py
##########
@@ -42,6 +42,29 @@
LocalStorageDriver = None
+class PickleableAcquireLockInSubprocess:
Review comment:
Yeah, IIRC there is a known issue with some Python 3.x version and pickling in some multiprocessing scenarios on OS X (encounter this issue in a different project).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on a change in pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
Kami commented on a change in pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#discussion_r825423677
##########
File path: libcloud/test/storage/test_local.py
##########
@@ -42,6 +42,29 @@
LocalStorageDriver = None
+class PickleableAcquireLockInSubprocess:
Review comment:
Yeah, IIRC there is a known issue with some Python 3.x version and pickling in some multiprocessing scenarios on OS X (encounter this issue in a different project).
EDIT: https://github.com/spack/spack/issues/14102
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
Kami commented on pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#issuecomment-1077518607
And yet another short term option would be to use pre-built artifact from https://github.com/apache/libcloud/actions/runs/2033897262.
But since those builds artifacts are not stored permanently that means you would need to bundle it with your app git repo or similar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
Kami commented on pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#issuecomment-1070581864
@Germandrummer92 will try to do it this week, but sadly can't promise anything.
Having said that, if you can't install it directly from git using pip, could you perhaps install it using pip from zip archive - https://github.com/apache/libcloud/archive/8c4f229c98f521281a468070507f5387f6255bef.zip?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Germandrummer92 commented on a change in pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
Germandrummer92 commented on a change in pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#discussion_r825299410
##########
File path: libcloud/test/storage/test_local.py
##########
@@ -42,6 +42,29 @@
LocalStorageDriver = None
+class PickleableAcquireLockInSubprocess:
Review comment:
I am not sure about this one, for me the local test was failing with "unpickleable" errors, I am not sure how it worked before/on CI? Maybe it's also just a mac problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Germandrummer92 commented on pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
Germandrummer92 commented on pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#issuecomment-1065879741
@RunOrVeith @Kami fyi would be great to get this merged/released
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Germandrummer92 commented on pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
Germandrummer92 commented on pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#issuecomment-1068931986
sorry to bothe you again @Kami: but any chance we can get a patch release with this change? It's proving surprisingly difficult on our side unfortunately to introduce a git dependency due to some poetry intricacies it turns out, an official release would make this much easier
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
Kami commented on pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#issuecomment-1077696202
That's rather weird since Libcloud doesn't directly depend on ``cryptography``? Or is it a transitive dependency of the ``requests`` library or similar?
In any case, we will try to do a patch release as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Germandrummer92 commented on pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
Germandrummer92 commented on pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#issuecomment-1071360638
We managed now, thanks anyways!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
Kami commented on pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#issuecomment-1072862966
Glad to hear you sorted out.
One thing we could still do is to build dev .tar.gz and .whl bundle on each push to master and upload it as an artifact as a part of Github Actions build.
That wouldn't be for long term production use, but mostly for testing and similar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
Kami commented on pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#issuecomment-1077484474
@RunOrVeith I will work on it this weekend.
Having said that, how about installing it from a .zip / .tar.gz bundle as proposed above? I quickly tested it locally in isolation and it seems to be working fine.
```toml
[tool.poetry.dependencies]
python = "^3.8"
poetry = "init"
apache-libcloud = {url = "https://github.com/apache/libcloud/archive/8c4f229c98f521281a468070507f5387f6255bef.zip"}
```
Also pretty curious why poetry git install takes so long. Libcloud repository is small and even if poetry doesn't use a shallow clone (which it appears it doesn't), cloning the repo should be fast. I haven't installed git dependencies using poetry yet, but I installed tons of dependency from git directly via pip without issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
Kami commented on pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#issuecomment-1066068405
Thanks for the contribution.
The change seems reasonable to me and I think it should be safe to merge it as long as ll the tests pass :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on pull request #1664: Enable Retrying of Raw Requests parse_error
Posted by GitBox <gi...@apache.org>.
Kami commented on pull request #1664:
URL: https://github.com/apache/libcloud/pull/1664#issuecomment-1067751310
Merged, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org