You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "akitoshka (via GitHub)" <gi...@apache.org> on 2023/07/26 11:35:27 UTC
[GitHub] [kafka] akitoshka opened a new pull request, #14101: Source task stop call was added to force stopping execution.
akitoshka opened a new pull request, #14101:
URL: https://github.com/apache/kafka/pull/14101
We face a problem with restarting of the tasks, sometimes it leads to resource leak.
We used the jdbc source connector and noticed an increasing of count of opened sessions on Vertica side. But this problem is applicable for all databases and possibly for all source connectors.
Our case is the next:
1) Run jdbc source connector (io.confluent.connect.jdbc.JdbcSourceConnector) and set poll.interval.ms (86400000) > task.shutdown.graceful.timeout.ms (it's the property on Kafka-connect side, we set 10000)
2) Send POST /connectors/<connector_name>/tasks/<task_number>/restart
ER: count of session is the same as before restart
AR: count of session increases
The main problem is when org.apache.kafka.connect.runtime.Worker#stopAndAwaitTasks(java.util.Collection<org.apache.kafka.connect.util.ConnectorTaskId>) method is called it doesn't stop a source task itself.
The source task stops only if polling process stops on source task side. But if polling is continuing for a long time it leads to resource occupation as a result it can lead to the fail on DB side. We suggest to stop source task explicitly
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Source task stop call was added to force stopping execution. [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #14101: Source task stop call was added to force stopping execution.
URL: https://github.com/apache/kafka/pull/14101
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vamossagar12 commented on pull request #14101: Source task stop call was added to force stopping execution.
Posted by "vamossagar12 (via GitHub)" <gi...@apache.org>.
vamossagar12 commented on PR #14101:
URL: https://github.com/apache/kafka/pull/14101#issuecomment-1651871848
hey @akitoshka thanks for the PR. One thing that you have rightly pointed out is that `stopAndAwaitTasks` doesn't directly trigger a task stoppage. The restart request is handled by `DistributedHerder` in it's tick thread [here](https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L472). This will trigger the task stop [here](https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1475) and this will ask the task threads a signal to shutdown by setting the stopping [flag to true](https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerTask.java#L114). Note that the herder thread is a different one from the task thread thereby making the process asynchronous. And all the tasks need to shutdown within 10s in your case or else we will see messages like `Gra
ceful Shutdown failed`
Now the source tasks will , in the start of execute loop figure out that they have been asked to [stop](https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L340C26-L340C26) and it exits from the loop. This will force a close to be triggered [here](https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerTask.java#L216) which eventually leads to the closure of task [here](https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L314)
You might already know all this haha but just thought i will re-iterate the flow here. At this point I am not totally sure if we should add an explicit task stop here because the framework does handle invoking the task stop. Maybe in your case, increasing with the timeouts might help?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vamossagar12 commented on pull request #14101: Source task stop call was added to force stopping execution.
Posted by "vamossagar12 (via GitHub)" <gi...@apache.org>.
vamossagar12 commented on PR #14101:
URL: https://github.com/apache/kafka/pull/14101#issuecomment-1655210284
> This is exactly our case: we have long source polling (in our case it's jdbc source task), each restarting doesn't break previous task (we get Graceful Shutdown failed), but creates new one => a lot of connections on db side.
Yeah that's what I had imagined would be happening in your case. As I said you will need to play around with the graceful shutdown timeout and probably with the poll interval on the JDBC connector side (that would be out of scope here). IMO it might be hard to factor in the change in this PR considering that meddles with the current logic and we will need to make more changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Source task stop call was added to force stopping execution. [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14101:
URL: https://github.com/apache/kafka/pull/14101#issuecomment-2563280611
This PR has been closed since it has not had any activity in 120 days. If you feel like this
was a mistake, or you would like to continue working on it, please feel free to re-open the
PR and ask for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vamossagar12 commented on pull request #14101: Source task stop call was added to force stopping execution.
Posted by "vamossagar12 (via GitHub)" <gi...@apache.org>.
vamossagar12 commented on PR #14101:
URL: https://github.com/apache/kafka/pull/14101#issuecomment-1700372474
@akitoshka , Going by all these explanations above, do you think we can close this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] akitoshka commented on pull request #14101: Source task stop call was added to force stopping execution.
Posted by "akitoshka (via GitHub)" <gi...@apache.org>.
akitoshka commented on PR #14101:
URL: https://github.com/apache/kafka/pull/14101#issuecomment-1653207380
@vamossagar12 and @yashmayya thank you very much for your detailed comments
@vamossagar12 you are absolutely right about the flow.
> Now the source tasks will , in the start of execute loop figure out that they have been asked to [stop](https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L340C26-L340C26) and it exits from the loop.
One comment here: the source task will figure out that they have been asked to stop, only after [polling](https://github.com/apache/kafka/blob/d2fc9076231261db947168776505f0c21418a25f/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L357) is stopped. In general, polling can be a long process so we suggested to call task.stop explicitly and earlier to interrupt task polling.
@yashmayya
unfortunately, the mentioned mechanism doesn't break task forcely and doesn't close task resources after passing `task.shutdown.graceful.timeout.ms`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] C0urante commented on pull request #14101: Source task stop call was added to force stopping execution.
Posted by "C0urante (via GitHub)" <gi...@apache.org>.
C0urante commented on PR #14101:
URL: https://github.com/apache/kafka/pull/14101#issuecomment-1656533337
There's also [KAFKA-15090](https://issues.apache.org/jira/browse/KAFKA-15090), which gives some context on why we stopped invoking `SourceTask::stop` on the herder thread (which is the thread that would be invoking that method in this PR). I've been exploring some potential fixes, will publish a PR next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] yashmayya commented on pull request #14101: Source task stop call was added to force stopping execution.
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on PR #14101:
URL: https://github.com/apache/kafka/pull/14101#issuecomment-1652085859
Adding to what Sagar said above - this whole machinery is intentional because there is a mechanism [here](https://github.com/apache/kafka/blob/6d81698ba4cb9c62185b0a50cc81ddea3047994d/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L1075-L1083) to cancel / abandon tasks that can't be stopped within `task.shutdown.graceful.timeout.ms` (worker configuration) and avoid blocking indefinitely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] gharris1727 commented on pull request #14101: Source task stop call was added to force stopping execution.
Posted by "gharris1727 (via GitHub)" <gi...@apache.org>.
gharris1727 commented on PR #14101:
URL: https://github.com/apache/kafka/pull/14101#issuecomment-1656189960
Hey thanks @akitoshka for giving some attention to this issue.
Unfortunately the fix as-is can't be merged, for the reasons that @yashmayya and @vamossagar12 raised above. While the particular plugin behavior (resource leaks) is out-of-scope for the framework to fix, I do think it would be appropriate for the framework to indicate to plugins that they have been cancelled, to allow themselves to clean up.
There is a relevant ticket here: https://issues.apache.org/jira/browse/KAFKA-14725 which covers some of the suggested fixes, and I'm happy to review a PR for that.
Thanks again for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Source task stop call was added to force stopping execution. [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14101:
URL: https://github.com/apache/kafka/pull/14101#issuecomment-1833053201
This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please ask a committer for review. If the PR has merge conflicts, please update it with the latest from trunk (or appropriate release branch) <p> If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org