You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/29 17:03:37 UTC
[GitHub] [beam] lostluck opened a new issue, #23422: [Task][Go SDK]: Dataflow Go PostCommits timing out
lostluck opened a new issue, #23422:
URL: https://github.com/apache/beam/issues/23422
### What needs to happen?
Currently the postcommits are failing. We need to investigate where the time is being spent and if possible, either parallelized it better (WRT beam project Dataflow quotas), or consolidate some test pipelines into single jobs (better handling quota), or reduce the number of specific tests we're running against dataflow post commits (eg. remove low signal tests).
As a stop gap, double the 2.5h timeout to 5h.
### Issue Priority
Priority: 1
### Issue Component
Component: sdk-go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on issue #23422: [Task][Go SDK]: Dataflow Go PostCommits timing out
Posted by GitBox <gi...@apache.org>.
lostluck commented on issue #23422:
URL: https://github.com/apache/beam/issues/23422#issuecomment-1262605572
Specifically, we could reduce the number of pipelines by consolidating them into a single pipeline for the specific area. The we have Dataflow filtering out the individual pipelines where the consolidated ones are used instead. This may require some test prefix renaming to simplify.
https://github.com/apache/beam/blob/a393efa61d4a1933717a4fe24a7559ec00e8912a/sdks/go/test/integration/integration.go#L222
eg. All the State Pipelines could be consolidated into a single pipeline:
TestConsolidatedStateAPIPipeline
And the State tests renamed and prefixed, eg. TestStateAPI_MapState
And similarly across common areas in the integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on issue #23422: [Task][Go SDK]: Dataflow Go PostCommits timing out
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on issue #23422:
URL: https://github.com/apache/beam/issues/23422#issuecomment-1726610329
Looks like this has is stabilized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck closed issue #23422: [Task][Go SDK]: Dataflow Go PostCommits timing out
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck closed issue #23422: [Task][Go SDK]: Dataflow Go PostCommits timing out
URL: https://github.com/apache/beam/issues/23422
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on issue #23422: [Task][Go SDK]: Dataflow Go PostCommits timing out
Posted by GitBox <gi...@apache.org>.
lostluck commented on issue #23422:
URL: https://github.com/apache/beam/issues/23422#issuecomment-1262810771
Yeah, transient or not, it's a flake, which means it interrupts velocity.
Since our apache-beam-testing project for Dataflow is a shared resource and we have only so many simultaneous job slots, I'd rather get the post commit back down under an hour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #23422: [Task][Go SDK]: Dataflow Go PostCommits timing out
Posted by GitBox <gi...@apache.org>.
Abacn commented on issue #23422:
URL: https://github.com/apache/beam/issues/23422#issuecomment-1262794123
Seen this before and was transient #23311
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org