You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2023/05/10 03:16:56 UTC
[GitHub] [beam] Abacn opened a new issue, #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Abacn opened a new issue, #26621:
URL: https://github.com/apache/beam/issues/26621
### What happened?
Since Apr 25, 2023.
Error massage:
```
23:05:17 org.apache.beam.sdk.io.sparkreceiver.SparkReceiverIOIT > testSparkReceiverIOReadsInStreamingWithOffset FAILED
23:05:17 java.lang.AssertionError: expected:<5000000> but was:<4700000>
23:05:17 at org.junit.Assert.fail(Assert.java:89)
23:05:17 at org.junit.Assert.failNotEquals(Assert.java:835)
23:05:17 at org.junit.Assert.assertEquals(Assert.java:647)
23:05:17 at org.junit.Assert.assertEquals(Assert.java:633)
23:05:17 at org.apache.beam.sdk.io.sparkreceiver.SparkReceiverIOIT.testSparkReceiverIOReadsInStreamingWithOffset(SparkReceiverIOI
```
the actual number of element is fluctuating and always smaller than the expected. May indicates a data loss bug
### Issue Failure
Failure: Test is continually failing
### Issue Priority
Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26621:
URL: https://github.com/apache/beam/issues/26621#issuecomment-1557293652
Thanks, changes should be handy, opened #26621
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] riteshghorse commented on issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on issue #26621:
URL: https://github.com/apache/beam/issues/26621#issuecomment-1551902707
Looking at the issue https://github.com/apache/beam/pull/26051, it closed https://github.com/apache/beam/issues/26011 which was P3 issue. As per our release blocker policy https://beam.apache.org/contribute/release-blocking/ I believe we can move this to 2.49.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26621:
URL: https://github.com/apache/beam/issues/26621#issuecomment-1552336777
Surprisingly the test is back green yesterday: https://ci-beam.apache.org/view/PerformanceTests/job/beam_PerformanceTests_SparkReceiver_IO/ We can close it for now
Why it was recovered itself remains unclear...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26621:
URL: https://github.com/apache/beam/issues/26621#issuecomment-1541293362
Possible cause: #26051 the only SparkreceiverIO PR merged at that time （among 3 java sdk and runner change on that day)
even though the test was succeeded in trigger run: https://ci-beam.apache.org/job/beam_PerformanceTests_SparkReceiver_IO/379/
CC: @aromanenko-dev
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Amar3tto commented on issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on issue #26621:
URL: https://github.com/apache/beam/issues/26621#issuecomment-1556698299
On the Grafana dashboard, I can see that the last successful tests took about 27 minutes. This is pretty close to the timeout value = 30 minutes. We can increase the timeout to see if that helps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on issue #26621:
URL: https://github.com/apache/beam/issues/26621#issuecomment-1556642158
Is it just a flaky test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Amar3tto commented on issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on issue #26621:
URL: https://github.com/apache/beam/issues/26621#issuecomment-1556709641
I'll create a new PR with a different timeout.
I will also try to optimize the test so that it takes less time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26621:
URL: https://github.com/apache/beam/issues/26621#issuecomment-1553412332
It's failing again after 3 successful run, same reason (not all record counted). Reopen...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26621:
URL: https://github.com/apache/beam/issues/26621#issuecomment-1551782603
Sorry #26738 does not disable this test. I linked wrong issue at that PR. This is a performance test not in the validation postcommits for the release so up to @riteshghorse to determine whether it's a blocker. If not made into 2.48.0 we should either revert #26051 or release with a known issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] riteshghorse commented on issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on issue #26621:
URL: https://github.com/apache/beam/issues/26621#issuecomment-1551738993
+1 to @Abacn 's PR to disable the test for now and target this for 2.49.0 Release
Let me know if you think otherwise @Amar3tto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] riteshghorse commented on issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on issue #26621:
URL: https://github.com/apache/beam/issues/26621#issuecomment-1551814309
okay, Saw disabling the test and stopped there. Thanks for catching Yi!
@aromanenko-dev what do you think about moving this to 2.49.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Amar3tto commented on issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on issue #26621:
URL: https://github.com/apache/beam/issues/26621#issuecomment-1548990264
> CC: @Amar3tto https://ci-beam.apache.org/job/beam_PerformanceTests_SparkReceiver_IO/ is failing. Do you have any ideas how it can be fixed?
Working on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn closed issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
URL: https://github.com/apache/beam/issues/26621
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26621:
URL: https://github.com/apache/beam/issues/26621#issuecomment-1541327938
#23785 passed on test branch by reverting a14f98a
CC: @Amar3tto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on issue #26621:
URL: https://github.com/apache/beam/issues/26621#issuecomment-1547763886
CC: @Amar3tto
https://ci-beam.apache.org/job/beam_PerformanceTests_SparkReceiver_IO/ is failing.
Do you have any ideas how it can be fixed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26621:
URL: https://github.com/apache/beam/issues/26621#issuecomment-1557237490
I took a look again and the cause might be here:
https://github.com/apache/beam/blob/120f9525177c12bc7d282d021578303b29ed452e/.test-infra/jenkins/job_PerformanceTests_SparkReceiverIO_IT.groovy#L70
Previously the number of worker was 5 (as other performance tests) and #26051 changed it to 1. The test may simply not reach its end by timeout.
From the dashboard it takes >27 min http://104.154.241.245/d/bnlHKP3Wz/java-io-it-tests-dataflow?orgId=1&viewPanel=27 to complete when succeeded, very close to timeout (30 min)
Shall we just change back numWorkers to 5 ? @Amar3tto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on issue #26621:
URL: https://github.com/apache/beam/issues/26621#issuecomment-1557268135
Also, since it takes quite significant amount of time to finish, I'd recommend to run this job less often - once per day, for example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev closed issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev closed issue #26621: [Failing Test]: beam_PerformanceTests_SparkReceiver_IO failing
URL: https://github.com/apache/beam/issues/26621
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org