You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/23 00:47:48 UTC
[GitHub] [beam] aaltay opened a new issue, #22008: [Bug]: org.apache.beam.sdk.io.gcp.spanner.SpannerIOWriteExceptionHandlingTest.testExceptionHandlingForWriteGrouped flaky
aaltay opened a new issue, #22008:
URL: https://github.com/apache/beam/issues/22008
### What happened?
Example: https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/5316/
@johnjcasey - Who could own this issue?
### Issue Priority
Priority: 1
### Issue Component
Component: io-java-gcp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem closed issue #22008: [Bug]: org.apache.beam.sdk.io.gcp.spanner.SpannerIOWriteExceptionHandlingTest.testExceptionHandlingForWriteGrouped flaky
Posted by GitBox <gi...@apache.org>.
pabloem closed issue #22008: [Bug]: org.apache.beam.sdk.io.gcp.spanner.SpannerIOWriteExceptionHandlingTest.testExceptionHandlingForWriteGrouped flaky
URL: https://github.com/apache/beam/issues/22008
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on issue #22008: [Bug]: org.apache.beam.sdk.io.gcp.spanner.SpannerIOWriteExceptionHandlingTest.testExceptionHandlingForWriteGrouped flaky
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #22008:
URL: https://github.com/apache/beam/issues/22008#issuecomment-1163940464
> Sure, it's in [FluentBackoff.java](https://github.com/apache/beam/blob/52e1b3ff9e9b1c5067b59bbb8053f0bc53f9679f/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FluentBackoff.java#L193). You'll note that there's also a bit of randomness added in each step so what could very well be happening is that sometimes it maxes out at 9 retries and other times it maxes out at 10. With no randomness it would always hit 10 I think. It was failing pretty consistently for me in local testing.
>
> That code also has things like max retries, but they don't come into play because the defaults are high (INT_MAX and 1000 days of cumulative backlog time).
>
> Assuming this test really does need to care about retries it seems like there are several fairly straightforward options:
>
> 1. Test for retries > 1 instead of a specific number. This means changes to those defaults would go uncaught, but not sure why this particular test should care about the specific semantics of the backoff code (which has its own tests)
> 2. Explicitly set the max retries and an appropriate cumulative backoff to go with it
> 3. Seed Random's RNG with a fixed value before the test and use the number of retries consistent with that RNG value (if it is indeed switching between 9 and 10)
Ack. All 3 sounds like reasonable options.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] byronellis commented on issue #22008: [Bug]: org.apache.beam.sdk.io.gcp.spanner.SpannerIOWriteExceptionHandlingTest.testExceptionHandlingForWriteGrouped flaky
Posted by GitBox <gi...@apache.org>.
byronellis commented on issue #22008:
URL: https://github.com/apache/beam/issues/22008#issuecomment-1163822044
Oh! I was looking at this one. I was trying to decide if it was an off-by-one error. It looks like the default back off formula is 15000*1.5^k and I think k=10 would be the last retry for a max back off of 900000?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] byronellis commented on issue #22008: [Bug]: org.apache.beam.sdk.io.gcp.spanner.SpannerIOWriteExceptionHandlingTest.testExceptionHandlingForWriteGrouped flaky
Posted by GitBox <gi...@apache.org>.
byronellis commented on issue #22008:
URL: https://github.com/apache/beam/issues/22008#issuecomment-1164844591
.take-issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] byronellis commented on issue #22008: [Bug]: org.apache.beam.sdk.io.gcp.spanner.SpannerIOWriteExceptionHandlingTest.testExceptionHandlingForWriteGrouped flaky
Posted by GitBox <gi...@apache.org>.
byronellis commented on issue #22008:
URL: https://github.com/apache/beam/issues/22008#issuecomment-1163910489
Sure, it's in [FluentBackoff.java](https://github.com/apache/beam/blob/52e1b3ff9e9b1c5067b59bbb8053f0bc53f9679f/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FluentBackoff.java#L193). You'll note that there's also a bit of randomness added in each step so what could very well be happening is that sometimes it maxes out at 9 retries and other times it maxes out at 10. With no randomness it would always hit 10 I think. It was failing pretty consistently for me in local testing.
That code also has things like max retries, but they don't come into play because the defaults are high (INT_MAX and 1000 days of cumulative backlog time).
Assuming this test really does need to care about retries it seems like there are several fairly straightforward options:
1. Test for retries > 1 instead of a specific number. This means changes to those defaults would go uncaught, but not sure why this particular test should care about the specific semantics of the backoff code (which has its own tests)
2. Explicitly set the max retries and an appropriate cumulative backoff to go with it
3. Seed Random's RNG with a fixed value before the test and use the number of retries consistent with that RNG value (if it is indeed switching between 9 and 10)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on issue #22008: [Bug]: org.apache.beam.sdk.io.gcp.spanner.SpannerIOWriteExceptionHandlingTest.testExceptionHandlingForWriteGrouped flaky
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #22008:
URL: https://github.com/apache/beam/issues/22008#issuecomment-1163879700
> Oh! I was looking at this one. I was trying to decide if it was an off-by-one error. It looks like the default back off formula is 15000*1.5^k and I think k=10 would be the last retry for a max back off of 900000?
I agree, that formula sounds off.
Could you add a code link? I am not sure where is that backoff code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org