You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/04/06 12:16:48 UTC
[GitHub] [pulsar] Esakkimuthu991 opened a new issue #6675: Kafka Source
Connector in pulsar cluster
Esakkimuthu991 opened a new issue #6675: Kafka Source Connector in pulsar cluster
URL: https://github.com/apache/pulsar/issues/6675
**Describe the bug**
Exception in kafka source connector in pulsar cluster function worker
**To Reproduce**
Steps to reproduce the behavior:
1. Run Pulsar Cluster - one zookeeper, one broker and three bookie
2. Ensure Function worker enabled
3. Create Source for kafka
4. Running kafka container and produce message to it.
5. No Exception thrown instead getting hanged . while checking logs no more meaningful errors
**Expected behavior**
Running Kafka Source Connector through function worker mode in pulsar cluster
**Screenshots**
**Desktop (please complete the following information):**
- OS: Windows
**Additional context**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on issue #6675: Kafka Source Connector in
pulsar cluster
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #6675: Kafka Source Connector in pulsar cluster
URL: https://github.com/apache/pulsar/issues/6675#issuecomment-611113081
@Esakkimuthu991 do you mean a docker-compose file for cluster setup?
For step-by-step instruction on setting up a cluster using docker, you can check out this demo:
https://github.com/streamnative/tgip/blob/master/episodes/001/demo.md
I walk through it in this TGIP video: https://www.youtube.com/watch?v=1RQSot5tTuU
Hope this is useful for you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] Esakkimuthu991 commented on issue #6675: Kafka Source
Connector in pulsar cluster
Posted by GitBox <gi...@apache.org>.
Esakkimuthu991 commented on issue #6675: Kafka Source Connector in pulsar cluster
URL: https://github.com/apache/pulsar/issues/6675#issuecomment-610183030
> @Esakkimuthu991
>
> What version of the Kafka cluster are you running? What version of Pulsar are you using?
pulsar image - apachepulsar/pulsar-test-latest-version:latest
kafka image - spotify/kafka:latest
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on issue #6675: Kafka Source Connector in
pulsar cluster
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #6675: Kafka Source Connector in pulsar cluster
URL: https://github.com/apache/pulsar/issues/6675#issuecomment-609935815
@Esakkimuthu991
What version of the Kafka cluster are you running? What version of Pulsar are you using?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] Esakkimuthu991 commented on issue #6675: Kafka Source
Connector in pulsar cluster
Posted by GitBox <gi...@apache.org>.
Esakkimuthu991 commented on issue #6675: Kafka Source Connector in pulsar cluster
URL: https://github.com/apache/pulsar/issues/6675#issuecomment-610304631
> There are a couple of things to check:
>
> 1. I am not sure what version of Kafka is spotify/kafka. I think the Kafka connector is using Kafka version 0.10.2.1. You need to make sure your Kafka cluster for testing is compatible with this version.
> 2. how do you configure `bootstrapServers` in the Kafka source connector? Is it reachable in pulsar cluster?
Earlier I tried the same spotify/kafka with pulsar standalone mode . Its supporting and source connector working fine.
Do we have docker compose file for pulsar cluster running with pulsar 2.5.0 image?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie closed issue #6675: Kafka Source Connector in pulsar
cluster
Posted by GitBox <gi...@apache.org>.
sijie closed issue #6675: Kafka Source Connector in pulsar cluster
URL: https://github.com/apache/pulsar/issues/6675
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on issue #6675: Kafka Source Connector in
pulsar cluster
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #6675: Kafka Source Connector in pulsar cluster
URL: https://github.com/apache/pulsar/issues/6675#issuecomment-610287164
There are a couple of things to check:
1) I am not sure what version of Kafka is spotify/kafka. I think the Kafka connector is using Kafka version 0.10.2.1. You need to make sure your Kafka cluster for testing is compatible with this version.
2) how do you configure `bootstrapServers` in the Kafka source connector? Is it reachable in pulsar cluster?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on issue #6675: Kafka Source Connector in
pulsar cluster
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #6675: Kafka Source Connector in pulsar cluster
URL: https://github.com/apache/pulsar/issues/6675#issuecomment-610529957
@Esakkimuthu991 you can use this docker-compose file for reference. https://github.com/apache/pulsar/blob/master/docker-compose/standalone-dashboard/docker-compose.yml
I think the important thing is to make sure your function worker is able to connect to your Kafka cluster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on issue #6675: Kafka Source Connector in
pulsar cluster
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #6675: Kafka Source Connector in pulsar cluster
URL: https://github.com/apache/pulsar/issues/6675#issuecomment-611732642
@Esakkimuthu991 I am assuming you have gotten the information you need. I am closing this issue now. If you need any more help, feel free to leave a comment and reopen this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] Esakkimuthu991 commented on issue #6675: Kafka Source
Connector in pulsar cluster
Posted by GitBox <gi...@apache.org>.
Esakkimuthu991 commented on issue #6675: Kafka Source Connector in pulsar cluster
URL: https://github.com/apache/pulsar/issues/6675#issuecomment-611478728
> @Esakkimuthu991 do you mean a docker-compose file for cluster setup?
>
> For step-by-step instruction on setting up a cluster using docker, you can check out this demo:
>
> https://github.com/streamnative/tgip/blob/master/episodes/001/demo.md
>
> I walk through it in this TGIP video: https://www.youtube.com/watch?v=1RQSot5tTuU
>
> Hope this is useful for you.
Thanks for the link provided
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] Esakkimuthu991 commented on issue #6675: Kafka Source
Connector in pulsar cluster
Posted by GitBox <gi...@apache.org>.
Esakkimuthu991 commented on issue #6675: Kafka Source Connector in pulsar cluster
URL: https://github.com/apache/pulsar/issues/6675#issuecomment-610937650
Trying to run it on pulsar cluster mode . So it would be helpful if sharing any reference for pulsar cluster 2.5.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] Esakkimuthu991 commented on issue #6675: Kafka Source
Connector in pulsar cluster
Posted by GitBox <gi...@apache.org>.
Esakkimuthu991 commented on issue #6675: Kafka Source Connector in pulsar cluster
URL: https://github.com/apache/pulsar/issues/6675#issuecomment-610936347
> @Esakkimuthu991 you can use this docker-compose file for reference. https://github.com/apache/pulsar/blob/master/docker-compose/standalone-dashboard/docker-compose.yml
>
> I think the important thing is to make sure your function worker is able to connect to your Kafka cluster.
I've already tried with standalone mode pulsar and it is working
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services