You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "gabrywu (via GitHub)" <gi...@apache.org> on 2023/10/21 03:12:54 UTC
[PR] Involve CheckStopReadingFn class adding a chance to release resources gracefully [beam]
gabrywu opened a new pull request, #29100:
URL: https://github.com/apache/beam/pull/29100
closing #29072
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Involve CheckStopReadingFn class adding a chance to release resources gracefully [beam]
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on PR #29100:
URL: https://github.com/apache/beam/pull/29100#issuecomment-1791780799
@Abacn please help to review it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Involve CheckStopReadingFn class adding a chance to release resources gracefully [beam]
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on PR #29100:
URL: https://github.com/apache/beam/pull/29100#issuecomment-1809724264
> Thanks. The only caveat I could imagine is that this would break existing users using `SerializableFunction<TopicPartition, Boolean>` to `setCheckStopReadingFn`. It would be nice that setCheckStopReadingFn could still accept ordinary SerializableFunction
>
> also aware that there is no guarantee that tearDown will be called after a DoFn is done according to DoFnLifeCycle
can you give more details of this case? I find that parts of codes within tearDown run actually. Why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Involve CheckStopReadingFn class adding a chance to release resources gracefully [beam]
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on PR #29100:
URL: https://github.com/apache/beam/pull/29100#issuecomment-1786332938
> This generally LGTM. I looked at the bug and it makes sense. I don't really have context on what the intended uses of "checkStopReadingFn" are anyhow.
Beside official doc tells intended uses, we use it to simulate kafka consumer group.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Involve CheckStopReadingFn class adding a chance to release resources gracefully [beam]
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on PR #29100:
URL: https://github.com/apache/beam/pull/29100#issuecomment-1782616951
hi, @Abacn can you help to review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Involve CheckStopReadingFn class adding a chance to release resources gracefully [beam]
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on PR #29100:
URL: https://github.com/apache/beam/pull/29100#issuecomment-1811684162
Yes, per my understanding, teardown is used to clean up resources. Because dip don't have corresponding close method like flink, which is used to cleanup buffered data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Involve CheckStopReadingFn class adding a chance to release resources gracefully [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #29100:
URL: https://github.com/apache/beam/pull/29100#issuecomment-1810301660
Generally it will run. In the case there is an Exception in DoFn or runner side, for example, or pipeline get cancelled, teardown may not run. This was mostly about do not rely on teardown for data integrity management. Clean up resources in teardown are fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Involve CheckStopReadingFn class adding a chance to release resources gracefully [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #29100:
URL: https://github.com/apache/beam/pull/29100#discussion_r1389596528
##########
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/ReadFromKafkaDoFnTest.java:
##########
@@ -432,7 +432,7 @@ public void testProcessElementWhenTopicPartitionIsStopped() throws Exception {
makeReadSourceDescriptor(consumer)
.toBuilder()
.setCheckStopReadingFn(
- new SerializableFunction<TopicPartition, Boolean>() {
+ new CheckStopReadingFn() {
Review Comment:
```suggestion
new SerializableFunction<TopicPartition, Boolean>() {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Involve CheckStopReadingFn class adding a chance to release resources gracefully [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #29100:
URL: https://github.com/apache/beam/pull/29100#issuecomment-1806019137
Test passed locally
Test | Duration | Result
-- | -- | --
testProcessElementWhenTopicPartitionIsStopped | 0.140s | passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Involve CheckStopReadingFn class adding a chance to release resources gracefully [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #29100:
URL: https://github.com/apache/beam/pull/29100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Involve CheckStopReadingFn class adding a chance to release resources gracefully [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29100:
URL: https://github.com/apache/beam/pull/29100#issuecomment-1805633041
Reminder, please take a look at this pr: @kennknowles @Abacn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Involve CheckStopReadingFn class adding a chance to release resources gracefully [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #29100:
URL: https://github.com/apache/beam/pull/29100#discussion_r1389451198
##########
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/ReadFromKafkaDoFnTest.java:
##########
@@ -432,7 +432,7 @@ public void testProcessElementWhenTopicPartitionIsStopped() throws Exception {
makeReadSourceDescriptor(consumer)
.toBuilder()
.setCheckStopReadingFn(
- new SerializableFunction<TopicPartition, Boolean>() {
+ new CheckStopReadingFn() {
Review Comment:
sorry for being late. Let's revert the change in test to make sure the compatibility is tested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Involve CheckStopReadingFn class adding a chance to release resources gracefully [beam]
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on PR #29100:
URL: https://github.com/apache/beam/pull/29100#issuecomment-1786334204
> SerializableFunction<TopicPartition, Boolean>
OK, I will create a wrapper of SerializableFunction<TopicPartition, Boolean> to avoid breaking user experience.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Involve CheckStopReadingFn class adding a chance to release resources gracefully [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29100:
URL: https://github.com/apache/beam/pull/29100#issuecomment-1773650850
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @kennknowles for label java.
R: @Abacn for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org