You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "sudufly (via GitHub)" <gi...@apache.org> on 2023/05/12 01:57:18 UTC
[GitHub] [incubator-seatunnel] sudufly opened a new issue, #4735: [Bug] [Connector-kafka] consummer work abnormally when some partition dosen't has data
sudufly opened a new issue, #4735:
URL: https://github.com/apache/incubator-seatunnel/issues/4735
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
1. function pollNext waits for all partitions task to be done
but when some partitions dosen't has data, the CompletableFuture will blocked by the empty partition (poll(10000))
2. Is it necessary to do assign and seek everytime？
### SeaTunnel Version
2.3.1
### SeaTunnel Config
```conf
xx
```
### Running Command
```shell
xx
```
### Error Exception
```log
xx
```
### Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] liugddx commented on issue #4735: [Bug] [Connector-kafka] consummer work abnormally when some partition dosen't has data
Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on issue #4735:
URL: https://github.com/apache/incubator-seatunnel/issues/4735#issuecomment-1551209690
Do you mean the partition just quits without data? This is a streaming job, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] commented on issue #4735: [Bug] [Connector-kafka] consummer work abnormally when some partition dosen't has data
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4735:
URL: https://github.com/apache/seatunnel/issues/4735#issuecomment-1627880893
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] commented on issue #4735: [Bug] [Connector-kafka] consummer work abnormally when some partition dosen't has data
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4735:
URL: https://github.com/apache/seatunnel/issues/4735#issuecomment-1616218597
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] sudufly commented on issue #4735: [Bug] [Connector-kafka] consummer work abnormally when some partition dosen't has data
Posted by "sudufly (via GitHub)" <gi...@apache.org>.
sudufly commented on issue #4735:
URL: https://github.com/apache/incubator-seatunnel/issues/4735#issuecomment-1552343712
> Do you mean the partition just quits without data? This is a streaming job, right?
pollNext 是在一个while 里面执行的，这个方法会等待所有分区poll完成，但是在某种情况下，整个while会被阻塞（POLL_TIMEOUT 10s）
当某个分区没有数据（测试情况，或者某个分区先被消费完但是新数据还没上来），下一个while循环都会等待一个timeout的时间
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] sudufly commented on issue #4735: [Bug] [Connector-kafka] consummer work abnormally when some partition dosen't has data
Posted by "sudufly (via GitHub)" <gi...@apache.org>.
sudufly commented on issue #4735:
URL: https://github.com/apache/incubator-seatunnel/issues/4735#issuecomment-1552410636
就是在流模式下工作得不像一个流，像sparkStream ，或者自己写的kafka consumer,
就算只有一个分区有数据也能立马消费到数据，但是seatunnel 的 kafka source 在这种情况下 因为这个默认超时时间的缘故，表现就是每10s消费一次数据，如果所有分区都有数据的话就不会出现这个现象
> > > Do you mean the partition just quits without data? This is a streaming job, right?
> >
> >
> > pollNext 是在一个while 里面执行的，这个方法会等待所有分区poll完成，但是在某种情况下，整个while会被阻塞（POLL_TIMEOUT 10s） 当某个分区没有数据（测试情况，或者某个分区先被消费完但是新数据还没上来），下一个while循环都会等待一个timeout的时间
>
> I think this wait is necessary,What do you recommend?
就是在流模式下工作得不像一个流，像sparkStream ，或者自己写的kafka consumer,
就算只有一个分区有数据也能立马消费到数据，但是seatunnel 的 kafka source 在这种情况下 因为这个默认超时时间的缘故，表现就是每10s才能消费一次数据，如果所有分区都有数据的话就不会出现这个现象
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] liugddx commented on issue #4735: [Bug] [Connector-kafka] consummer work abnormally when some partition dosen't has data
Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on issue #4735:
URL: https://github.com/apache/incubator-seatunnel/issues/4735#issuecomment-1552355909
> > Do you mean the partition just quits without data? This is a streaming job, right?
>
> pollNext 是在一个while 里面执行的，这个方法会等待所有分区poll完成，但是在某种情况下，整个while会被阻塞（POLL_TIMEOUT 10s） 当某个分区没有数据（测试情况，或者某个分区先被消费完但是新数据还没上来），下一个while循环都会等待一个timeout的时间
I think this wait is necessary,What do you recommend?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] closed issue #4735: [Bug] [Connector-kafka] consummer work abnormally when some partition dosen't has data
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #4735: [Bug] [Connector-kafka] consummer work abnormally when some partition dosen't has data
URL: https://github.com/apache/seatunnel/issues/4735
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org