You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "aromanenko-dev (via GitHub)" <gi...@apache.org> on 2023/11/14 10:53:21 UTC
[I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]
aromanenko-dev opened a new issue, #29429:
URL: https://github.com/apache/beam/issues/29429
### What happened?
A Jenkins job [beam_PostCommit_Java_Nexmark_Flink](https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Flink/) keeps running forever and not finished properly (aborted after a limit of 4 hours). So it wastes the resources of `apache-beam-jenkins-16` machine.
### Issue Failure
Failure: Test is continually failing
### Issue Priority
Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [X] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1814660966
The breakage happens between 7b38f743f78812fd240e2245abc490e6cb8997ed (inclusive) and 7b38f743f78812fd240e2245abc490e6cb8997ed (exclusive), #28614 is the only related change CC: @jto @tvalentyn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]
Posted by "jto (via GitHub)" <gi...@apache.org>.
jto commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1814777192
I meant I'd like to run a specific run locally on my machine so I can quickly iterate on a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #29429: [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished
URL: https://github.com/apache/beam/issues/29429
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1814775168
Under your PR, comment "Run Flink Runner Nexmark Tests" then Jenkins will pick up the PR and run the test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1816701060
@jto Thanks for fixing this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]
Posted by "jto (via GitHub)" <gi...@apache.org>.
jto commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1816392479
Hey there! I made a PR to fix the issue: https://github.com/apache/beam/pull/29465
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]
Posted by "jto (via GitHub)" <gi...@apache.org>.
jto commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1814768397
Hey! I may have an idea but I'd need a way to run the specific test that's blocked to confirm. How can I do that ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1814712627
Comparing the log it seems the pipeline is executed differently.
Before:
```
INFO: Source: Query0.ReadUnbounded -> Flat Map -> Query0/Query0.Monitor/ParMultiDo(Monitor) -> Query0/Query0.Snoop/ParMultiDo(Anonymous) -> Query0/Query0/Query0.Serialize/ParMultiDo(Anonymous) -> Query0/Query0.Debug/ParMultiDo(Monitor) -> Query0/Query0.Stamp/ParMultiDo(Anonymous) -> Query0.Format/ParMultiDo(Anonymous) -> Query0.DevNull/ParMultiDo(Anonymous) (1/2) (b5360562850a3be1374b8798dfacfdce) switched from DEPLOYING to INITIALIZING.
Oct 27, 2023 4:25:11 PM org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper initializeState
INFO: No restore state for UnboundedSourceWrapper.
Oct 27, 2023 4:25:11 PM org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper initializeState
INFO: No restore state for UnboundedSourceWrapper.
Oct 27, 2023 4:25:11 PM org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper open
INFO: Unbounded Flink Source 1/2 is reading from sources: [UnboundedEventSource(0, 1000), UnboundedEventSource(2000, 3000), UnboundedEventSource(4000, 5000), UnboundedEventSource(6000, 7000), UnboundedEventSource(8000, 9000), UnboundedEventSource(10000, 11000), UnboundedEventSource(12000, 13000), UnboundedEventSource(14000, 15000), ...]
INFO: Source: Query0.ReadUnbounded -> Flat Map -> Query0/Query0.Monitor/ParMultiDo(Monitor) -> Query0/Query0.Snoop/ParMultiDo(Anonymous) -> Query0/Query0/Query0.Serialize/ParMultiDo(Anonymous) -> Query0/Query0.Debug/ParMultiDo(Monitor) -> Query0/Query0.Stamp/ParMultiDo(Anonymous) -> Query0.Format/ParMultiDo(Anonymous) -> Query0.DevNull/ParMultiDo(Anonymous) (2/2)#0 (6cb95a324b37d5272f769a25e93e5906) switched from INITIALIZING to RUNNING.
```
after:
```
INFO: Source: Query0.ReadUnbounded -> Flat Map -> Query0/Query0.Monitor/ParMultiDo(Monitor) -> Query0/Query0.Snoop/ParMultiDo(Anonymous) -> Query0/Query0/Query0.Serialize/ParMultiDo(Anonymous) -> Query0/Query0.Debug/ParMultiDo(Monitor) -> Query0/Query0.Stamp/ParMultiDo(Anonymous) -> Query0.Format/ParMultiDo(Anonymous) -> Query0.DevNull/ParMultiDo(Anonymous) (2/2) (eee384163299676ced9b2a87bf24bbca) switched from DEPLOYING to INITIALIZING.
Oct 27, 2023 10:22:48 PM org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.unbounded.FlinkUnboundedSourceReader start
WARNING: AutoWatermarkInterval is not set, watermarks won't be emitted.
...
INFO: Source: Query0.ReadUnbounded -> Flat Map -> Query0/Query0.Monitor/ParMultiDo(Monitor) -> Query0/Query0.Snoop/ParMultiDo(Anonymous) -> Query0/Query0/Query0.Serialize/ParMultiDo(Anonymous) -> Query0/Query0.Debug/ParMultiDo(Monitor) -> Query0/Query0.Stamp/ParMultiDo(Anonymous) -> Query0.Format/ParMultiDo(Anonymous) -> Query0.DevNull/ParMultiDo(Anonymous) (2/2) (eee384163299676ced9b2a87bf24bbca) switched from INITIALIZING to RUNNING.
Oct 27, 2023 10:22:48 PM org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.FlinkSourceSplitEnumerator assignSplitsAndLog
INFO: Assigned splits [[SplitIndex: 0, BeamSource: UnboundedEventSource(0, 1000)], [SplitIndex: 2, BeamSource: UnboundedEventSource(2000, 3000)], [SplitIndex: 4, BeamSource: UnboundedEventSource(4000, 5000)], [SplitIndex: 6, BeamSource: UnboundedEventSource(6000, 7000)], [SplitIndex: 8, BeamSource: UnboundedEventSource(8000, 9000)], [SplitIndex: 10, BeamSource: UnboundedEventSource(10000, 11000)], [SplitIndex: 12, BeamSource: UnboundedEventSource(12000, 13000)], [SplitIndex: 14, BeamSource:
```
In particular, before, UnboundedEventSource splits at INITIALIZING, log from `org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper open`
After, UnboundedEventSource split at RUNNING, log from `org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.FlinkSourceSplitEnumerator assignSplitsAndLog`
From #28614, the PR should only change batch, but it's breaking streaming. @jto could you please investigate? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1816697061
This may affect the 2.52.0 release to prevent a streaming pipeline end simultaneously. However in a production streaming pipeline it is not supposed to finish so it should fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1814815324
just a node, if compile locally it's good to be based on the latest master or cherrypick #29446, otherwise the compile is slow due to reason stated in that PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1816694234
Fixed by #29465
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1814802680
The equivalent gradle command is
```
./gradlew :sdks:java:testing:nexmark:run -Pnexmark.runner=:runners:flink:1.15 -Pnexmark.args="--manageResources=false --monitorJobs=true --resourceNameMode=QUERY_RUNNER_AND_MODE --exportSummaryToInfluxDB=false --influxRetentionPolicy=forever --suite=SMOKE --streamTimeout=60 --runner=FlinkRunner --streaming=true --queryLanguage=sql"
```
see https://beam.apache.org/documentation/sdks/java/testing/nexmark/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1814643274
Same for GItHub Action. Start timing out on Oct 28th: https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Flink.yml?page=4&query=event%3Aschedule
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org