You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "tustvold (via GitHub)" <gi...@apache.org> on 2024/01/05 11:08:05 UTC

[I] LocalStack Tests Flaky in CI [arrow-rs]

tustvold opened a new issue, #5283:
URL: https://github.com/apache/arrow-rs/issues/5283

   **Describe the bug**
   <!--
   A clear and concise description of what the bug is.
   -->
   
   * https://github.com/apache/arrow-rs/actions/runs/7420960113/job/20193342943
   * https://github.com/apache/arrow-rs/actions/runs/7420960113/job/20193515256
   * https://github.com/apache/arrow-rs/actions/runs/7420960113/job/20193722235
   
   My suspicion is the container is running out of memory or otherwise crashing, but I have not been able to reproduce this locally
   
   **To Reproduce**
   <!--
   Steps to reproduce the behavior:
   -->
   
   **Expected behavior**
   <!--
   A clear and concise description of what you expected to happen.
   -->
   
   **Additional context**
   <!--
   Add any other context about the problem here.
   -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] LocalStack Tests Flaky in CI [arrow-rs]

Posted by "bentsku (via GitHub)" <gi...@apache.org>.
bentsku commented on issue #5283:
URL: https://github.com/apache/arrow-rs/issues/5283#issuecomment-1977382274

   Okay I finally could reproduce and find which requests were triggering the issue:
   I could reproduce by creating a pretty resource-constrained environment (Docker image with 0.5 CPUs)
   
   It's a `GetObject` with `Content-Length` set to 1, with the key prefix `RACE-`
   Followed by a `PutObject` of length 2.
   The first call get the entirety of the data with an indicated content-length of 1, but gets `10` so it fails. 
   The reproducer is not stable at all, I just run `cargo test aws::tests::s3_test --features aws -- --exact` again and again and it can fail around 2 to 10% of the time with 0.5CPU. 
   
   I'll work on a fix now, might have an idea where this could come from. Thanks again for reporting this, I'm sure we will finally get to the end of this! 🤞 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] LocalStack Tests Flaky in CI [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5283:
URL: https://github.com/apache/arrow-rs/issues/5283#issuecomment-1878674724

   Filed an upstream issue as I'm somewhat at a loss for how to debug this further - https://github.com/localstack/localstack/issues/10003


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] LocalStack Tests Flaky in CI [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #5283: LocalStack Tests Flaky in CI
URL: https://github.com/apache/arrow-rs/issues/5283


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] LocalStack Tests Flaky in CI [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5283:
URL: https://github.com/apache/arrow-rs/issues/5283#issuecomment-1878648876

   Error in https://github.com/apache/arrow-rs/actions/runs/7422271983/job/20197235081?pr=5285
   
   ```
   Traceback (most recent call last):
     File "/opt/code/localstack/localstack/http/asgi.py", line 548, in handle_http
       await response.write(packet)
     File "/opt/code/localstack/localstack/http/asgi.py", line 302, in write
       await self.send({"type": "http.response.body", "body": data, "more_body": True})
     File "/opt/code/localstack/.venv/lib/python3.11/site-packages/hypercorn/protocol/http_stream.py", line 179, in app_send
       await self.send(
     File "/opt/code/localstack/.venv/lib/python3.11/site-packages/hypercorn/protocol/h11.py", line 136, in stream_send
       await self._send_h11_event(h11.Data(data=event.data))
     File "/opt/code/localstack/.venv/lib/python3.11/site-packages/hypercorn/protocol/h11.py", line 240, in _send_h11_event
       data = self.connection.send(event)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/opt/code/localstack/.venv/lib/python3.11/site-packages/h11/_connection.py", line 512, in send
       data_list = self.send_with_data_passthrough(event)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/opt/code/localstack/.venv/lib/python3.11/site-packages/h11/_connection.py", line 545, in send_with_data_passthrough
       writer(event, data_list.append)
     File "/opt/code/localstack/.venv/lib/python3.11/site-packages/h11/_writers.py", line 65, in __call__
       self.send_data(event.data, write)
     File "/opt/code/localstack/.venv/lib/python3.11/site-packages/h11/_writers.py", line 91, in send_data
       raise LocalProtocolError("Too much data for declared Content-Length")
   h11._util.LocalProtocolError: Too much data for declared Content-Length
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] LocalStack Tests Flaky in CI [arrow-rs]

Posted by "bentsku (via GitHub)" <gi...@apache.org>.
bentsku commented on issue #5283:
URL: https://github.com/apache/arrow-rs/issues/5283#issuecomment-1976700951

   Hi @tustvold, I've seen the update in the GH issue, I'm not sure updating will help, I don't remember doing a lot of changes here. I will have a look again and try to reproduce locally as much as I can, and try to merge a possible race condition fix soon. Will keep you updated! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] LocalStack Tests Flaky in CI [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5283:
URL: https://github.com/apache/arrow-rs/issues/5283#issuecomment-1987520546

   Fantastic news, I'll probably wait until `3.3` as the regular CI failures appear to have abated for now, but will pre-empt this and use a image sha should they return


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] LocalStack Tests Flaky in CI [arrow-rs]

Posted by "bentsku (via GitHub)" <gi...@apache.org>.
bentsku commented on issue #5283:
URL: https://github.com/apache/arrow-rs/issues/5283#issuecomment-1979751032

   Hello! The issue should now be fixed, I ran 200 times `aws::tests::s3_test` and finally could not reproduce it anymore with the linked PR. It, however, will only be available in the `latest` image (should be available in around 30 to 45 minutes), not sure if you'd be willing to use this tag. I believe `3.3` will be released end of March.
   
   Thank you again for your patience in this and your helps in uncovering the issue! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org