You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "aromanenko-dev (via GitHub)" <gi...@apache.org> on 2023/11/14 10:53:21 UTC

[I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]

aromanenko-dev opened a new issue, #29429:
URL: https://github.com/apache/beam/issues/29429

   ### What happened?
   
   A Jenkins job [beam_PostCommit_Java_Nexmark_Flink](https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Flink/) keeps running forever and not finished properly (aborted after a limit of 4 hours). So it wastes the resources of `apache-beam-jenkins-16` machine.
   
   ### Issue Failure
   
   Failure: Test is continually failing
   
   ### Issue Priority
   
   Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)
   
   ### Issue Components
   
   - [ ] Component: Python SDK
   - [ ] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [ ] Component: IO connector
   - [ ] Component: Beam YAML
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Spark Runner
   - [X] Component: Flink Runner
   - [ ] Component: Samza Runner
   - [ ] Component: Twister2 Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1814660966

   The breakage happens between 7b38f743f78812fd240e2245abc490e6cb8997ed (inclusive) and 7b38f743f78812fd240e2245abc490e6cb8997ed (exclusive), #28614 is the only related change CC: @jto @tvalentyn


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]

Posted by "jto (via GitHub)" <gi...@apache.org>.
jto commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1814777192

   I meant I'd like to run a specific run locally on my machine so I can quickly iterate on a fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #29429: [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished
URL: https://github.com/apache/beam/issues/29429


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1814775168

   Under your PR, comment "Run Flink Runner Nexmark Tests" then Jenkins will pick up the PR and run the test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]

Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1816701060

   @jto Thanks for fixing this!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]

Posted by "jto (via GitHub)" <gi...@apache.org>.
jto commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1816392479

   Hey there! I made a PR to fix the issue: https://github.com/apache/beam/pull/29465


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]

Posted by "jto (via GitHub)" <gi...@apache.org>.
jto commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1814768397

   Hey! I may have an idea but I'd need a way to run the specific test that's blocked to confirm. How can I do that ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1814712627

   Comparing the log it seems the pipeline is executed differently.
   
   Before:
   ```
   INFO: Source: Query0.ReadUnbounded -> Flat Map -> Query0/Query0.Monitor/ParMultiDo(Monitor) -> Query0/Query0.Snoop/ParMultiDo(Anonymous) -> Query0/Query0/Query0.Serialize/ParMultiDo(Anonymous) -> Query0/Query0.Debug/ParMultiDo(Monitor) -> Query0/Query0.Stamp/ParMultiDo(Anonymous) -> Query0.Format/ParMultiDo(Anonymous) -> Query0.DevNull/ParMultiDo(Anonymous) (1/2) (b5360562850a3be1374b8798dfacfdce) switched from DEPLOYING to INITIALIZING.
   Oct 27, 2023 4:25:11 PM org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper initializeState
   INFO: No restore state for UnboundedSourceWrapper.
   Oct 27, 2023 4:25:11 PM org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper initializeState
   INFO: No restore state for UnboundedSourceWrapper.
   Oct 27, 2023 4:25:11 PM org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper open
   INFO: Unbounded Flink Source 1/2 is reading from sources: [UnboundedEventSource(0, 1000), UnboundedEventSource(2000, 3000), UnboundedEventSource(4000, 5000), UnboundedEventSource(6000, 7000), UnboundedEventSource(8000, 9000), UnboundedEventSource(10000, 11000), UnboundedEventSource(12000, 13000), UnboundedEventSource(14000, 15000), ...]
   INFO: Source: Query0.ReadUnbounded -> Flat Map -> Query0/Query0.Monitor/ParMultiDo(Monitor) -> Query0/Query0.Snoop/ParMultiDo(Anonymous) -> Query0/Query0/Query0.Serialize/ParMultiDo(Anonymous) -> Query0/Query0.Debug/ParMultiDo(Monitor) -> Query0/Query0.Stamp/ParMultiDo(Anonymous) -> Query0.Format/ParMultiDo(Anonymous) -> Query0.DevNull/ParMultiDo(Anonymous) (2/2)#0 (6cb95a324b37d5272f769a25e93e5906) switched from INITIALIZING to RUNNING.
   ```
   
   after:
   ```
   INFO: Source: Query0.ReadUnbounded -> Flat Map -> Query0/Query0.Monitor/ParMultiDo(Monitor) -> Query0/Query0.Snoop/ParMultiDo(Anonymous) -> Query0/Query0/Query0.Serialize/ParMultiDo(Anonymous) -> Query0/Query0.Debug/ParMultiDo(Monitor) -> Query0/Query0.Stamp/ParMultiDo(Anonymous) -> Query0.Format/ParMultiDo(Anonymous) -> Query0.DevNull/ParMultiDo(Anonymous) (2/2) (eee384163299676ced9b2a87bf24bbca) switched from DEPLOYING to INITIALIZING.
   Oct 27, 2023 10:22:48 PM org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.unbounded.FlinkUnboundedSourceReader start
   WARNING: AutoWatermarkInterval is not set, watermarks won't be emitted.
   ...
   INFO: Source: Query0.ReadUnbounded -> Flat Map -> Query0/Query0.Monitor/ParMultiDo(Monitor) -> Query0/Query0.Snoop/ParMultiDo(Anonymous) -> Query0/Query0/Query0.Serialize/ParMultiDo(Anonymous) -> Query0/Query0.Debug/ParMultiDo(Monitor) -> Query0/Query0.Stamp/ParMultiDo(Anonymous) -> Query0.Format/ParMultiDo(Anonymous) -> Query0.DevNull/ParMultiDo(Anonymous) (2/2) (eee384163299676ced9b2a87bf24bbca) switched from INITIALIZING to RUNNING.
   Oct 27, 2023 10:22:48 PM org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.FlinkSourceSplitEnumerator assignSplitsAndLog
   INFO: Assigned splits [[SplitIndex: 0, BeamSource: UnboundedEventSource(0, 1000)], [SplitIndex: 2, BeamSource: UnboundedEventSource(2000, 3000)], [SplitIndex: 4, BeamSource: UnboundedEventSource(4000, 5000)], [SplitIndex: 6, BeamSource: UnboundedEventSource(6000, 7000)], [SplitIndex: 8, BeamSource: UnboundedEventSource(8000, 9000)], [SplitIndex: 10, BeamSource: UnboundedEventSource(10000, 11000)], [SplitIndex: 12, BeamSource: UnboundedEventSource(12000, 13000)], [SplitIndex: 14, BeamSource: 
   ```
   
   In particular, before, UnboundedEventSource splits at INITIALIZING, log from `org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper open`
   
   After, UnboundedEventSource split at RUNNING, log from `org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.FlinkSourceSplitEnumerator assignSplitsAndLog`
   
   From #28614, the PR should only change batch, but it's breaking streaming. @jto could you please investigate? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1816697061

   This may affect the 2.52.0 release to prevent a streaming pipeline end simultaneously. However in a production streaming pipeline it is not supposed to finish so it should fine


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1814815324

   just a node, if compile locally it's good to be based on the latest master or cherrypick #29446, otherwise the compile is slow due to reason stated in that PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1816694234

   Fixed by #29465


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1814802680

   The equivalent gradle command is
   ```
   ./gradlew :sdks:java:testing:nexmark:run -Pnexmark.runner=:runners:flink:1.15 -Pnexmark.args="--manageResources=false --monitorJobs=true --resourceNameMode=QUERY_RUNNER_AND_MODE  --exportSummaryToInfluxDB=false  --influxRetentionPolicy=forever --suite=SMOKE  --streamTimeout=60  --runner=FlinkRunner --streaming=true --queryLanguage=sql"
   ```
   see https://beam.apache.org/documentation/sdks/java/testing/nexmark/ 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #29429:
URL: https://github.com/apache/beam/issues/29429#issuecomment-1814643274

   Same for GItHub Action. Start timing out on Oct 28th: https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Flink.yml?page=4&query=event%3Aschedule


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org