You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/04/25 22:55:39 UTC
[GitHub] [beam] hengfengli opened a new pull request, #17461: [BEAM-12164]: fix the negative throughput issue
hengfengli opened a new pull request, #17461:
URL: https://github.com/apache/beam/pull/17461
Somehow, the getSize() can be a negative number which must come from the local throughput estimator. Therefore, we set the lower bound of the throughput to 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #17461: [BEAM-12164]: fix the negative throughput issue
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #17461:
URL: https://github.com/apache/beam/pull/17461#issuecomment-1109117776
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on pull request #17461: [BEAM-14365]: fix the negative throughput issue
Posted by GitBox <gi...@apache.org>.
y1chi commented on PR #17461:
URL: https://github.com/apache/beam/pull/17461#issuecomment-1109313692
Is it clear where the negative throughput value is coming from? and setting it to 0 is the right thing to do when it is negative?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #17461: [BEAM-12164]: fix the negative throughput issue
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #17461:
URL: https://github.com/apache/beam/pull/17461#issuecomment-1109117780
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #17461: [BEAM-14365]: fix the negative throughput issue
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #17461:
URL: https://github.com/apache/beam/pull/17461#issuecomment-1109434676
# [Codecov](https://codecov.io/gh/apache/beam/pull/17461?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#17461](https://codecov.io/gh/apache/beam/pull/17461?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (07f30d2) into [master](https://codecov.io/gh/apache/beam/commit/266005cfbfdd67f41a928b6323a9b0540a8c4e2b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (266005c) will **increase** coverage by `0.01%`.
> The diff coverage is `91.11%`.
> :exclamation: Current head 07f30d2 differs from pull request most recent head 32e5d22. Consider uploading reports for the commit 32e5d22 to get more accurate results
```diff
@@ Coverage Diff @@
## master #17461 +/- ##
==========================================
+ Coverage 73.91% 73.92% +0.01%
==========================================
Files 687 689 +2
Lines 90334 90397 +63
==========================================
+ Hits 66766 66827 +61
- Misses 22384 22386 +2
Partials 1184 1184
```
| Flag | Coverage Δ | |
|---|---|---|
| go | `49.87% <66.66%> (+<0.01%)` | :arrow_up: |
| python | `83.64% <92.85%> (+0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/17461?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/metrics/metrics.go](https://codecov.io/gh/apache/beam/pull/17461/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL21ldHJpY3MvbWV0cmljcy5nbw==) | `49.36% <0.00%> (ø)` | |
| [sdks/go/pkg/beam/core/runtime/harness/session.go](https://codecov.io/gh/apache/beam/pull/17461/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zZXNzaW9uLmdv) | `19.51% <ø> (ø)` | |
| [sdks/python/apache\_beam/transforms/core.py](https://codecov.io/gh/apache/beam/pull/17461/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb3JlLnB5) | `91.84% <ø> (ø)` | |
| [.../python/apache\_beam/ml/inference/sklearn\_loader.py](https://codecov.io/gh/apache/beam/pull/17461/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL3NrbGVhcm5fbG9hZGVyLnB5) | `92.68% <92.68%> (ø)` | |
| [sdks/go/pkg/beam/core/graph/fn.go](https://codecov.io/gh/apache/beam/pull/17461/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2ZuLmdv) | `79.23% <100.00%> (ø)` | |
| [sdks/go/pkg/beam/core/util/ioutilx/read.go](https://codecov.io/gh/apache/beam/pull/17461/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvaW91dGlseC9yZWFkLmdv) | `75.00% <100.00%> (+0.64%)` | :arrow_up: |
| [...python/apache\_beam/examples/complete/distribopt.py](https://codecov.io/gh/apache/beam/pull/17461/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZGlzdHJpYm9wdC5weQ==) | `98.56% <100.00%> (ø)` | |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/17461/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
| [setup.py](https://codecov.io/gh/apache/beam/pull/17461/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/ml/inference/api.py](https://codecov.io/gh/apache/beam/pull/17461/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2FwaS5weQ==) | `90.47% <0.00%> (ø)` | |
| ... and [2 more](https://codecov.io/gh/apache/beam/pull/17461/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/17461?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/17461?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [266005c...32e5d22](https://codecov.io/gh/apache/beam/pull/17461?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi merged pull request #17461: [BEAM-14365]: fix the negative throughput issue
Posted by GitBox <gi...@apache.org>.
y1chi merged PR #17461:
URL: https://github.com/apache/beam/pull/17461
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] hengfengli commented on pull request #17461: [BEAM-14365]: fix the negative throughput issue
Posted by GitBox <gi...@apache.org>.
hengfengli commented on PR #17461:
URL: https://github.com/apache/beam/pull/17461#issuecomment-1109346515
> Is it clear where the negative throughput value is coming from? and setting it to 0 is the right thing to do when it is negative?
@y1chi No, we are still not sure where the negative number comes from since it only happened after running over 2 days. We got the following error:
"java.lang.IllegalArgumentException: Expected size >= 0 but received -36590.86666666666. at org.apache.beam.sdk.transforms.reflect.ByteBuddyDoFnInvokerFactory$DefaultGetSize.validateSize(ByteBuddyDoFnInvokerFactory.java:438) at org.apache.beam.sdk.io.gcp.spanner.changestreams.dofn.ReadChangeStreamPartitionDoFn$DoFnInvoker.invokeGetSize(Unknown Source) at org.apache.beam.fn.harness.FnApiDoFnRunner.calculateRestrictionSize(FnApiDoFnRunner.java:1182) at org.apache.beam.fn.harness.FnApiDoFnRunner.trySplitForElementAndRestriction(FnApiDoFnRunner.java:1625) at org.apache.beam.fn.harness.FnApiDoFnRunner.access$1800(FnApiDoFnRunner.java:142) at org.apache.beam.fn.harness.FnApiDoFnRunner$SplittableFnDataReceiver.trySplit(FnApiDoFnRunner.java:1113) at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$SplittingMetricTrackingFnDataReceiver.trySplit(PCollectionConsumerRegistry.java:342) at org.apache.beam.fn.harness.BeamFnDataReadRunner.trySplit(BeamFnDataReadRunner.java:259) at org.
apache.beam.fn.harness.control.ProcessBundleHandler.trySplit(ProcessBundleHandler.java:688) at org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:151) at org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:116) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) "
Basically, it comes from `getSize()` in `ReadChangeStreamPartitionDoFn`:
https://github.com/apache/beam/blob/07f30d221e4b285b23b74c3509d77b62388b7bb4/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dofn/ReadChangeStreamPartitionDoFn.java#L159-L172
The remainingWork in Progress cannot be negative ([code](https://github.com/apache/beam/blob/07f30d221e4b285b23b74c3509d77b62388b7bb4/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/RestrictionTracker.java#L187-L192)). So the only possibility is from the `throughput`.
Setting it to 0 is the current effective way to fix the issue. Also, the throughput is not a critical part and does not impact the correctness of the connector. When the throughput is negative, setting it to 0 is acceptable.
In addition, I have been running a dataflow job with printing more logs to find out where the negative number actually comes from.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] thiagotnunes commented on pull request #17461: [BEAM-12164]: fix the negative throughput issue
Posted by GitBox <gi...@apache.org>.
thiagotnunes commented on PR #17461:
URL: https://github.com/apache/beam/pull/17461#issuecomment-1109138842
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org