You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/06/15 01:14:30 UTC
[GitHub] [pulsar] btsken opened a new issue #7273: Functions doesn't subscribe input topic
btsken opened a new issue #7273:
URL: https://github.com/apache/pulsar/issues/7273
**Describe the bug**
I create a function in k8s environment. I can see a pod is active. But I found the function doesn't subscribe to the input topic
I modified [jobNamespace ](https://github.com/apache/pulsar/blob/master/deployment/kubernetes/helm/pulsar/templates/broker-configmap.yaml#L64), [serviceUrl ](https://github.com/apache/pulsar/blob/master/deployment/kubernetes/helm/pulsar/templates/broker-configmap.yaml#L68)and [adminUrl](https://github.com/apache/pulsar/blob/master/deployment/kubernetes/helm/pulsar/templates/broker-configmap.yaml#L67), I set the URL to proxy url.
**To Reproduce**
Steps to reproduce the behavior:
1. modified PF_functionRuntimeFactoryConfigs_jobNamespace, PF_functionRuntimeFactoryConfigs_pulsarAdminUrl, PF_functionRuntimeFactoryConfigs_pulsarServiceUrl in [broker-configmap.yaml](https://github.com/apache/pulsar/blob/master/deployment/kubernetes/helm/pulsar/templates/broker-configmap.yaml)
`PF_functionRuntimeFactoryConfigs_jobNamespace ` is different to pulsar cluster
`PF_functionRuntimeFactoryConfigs_pulsarAdminUrl and PF_functionRuntimeFactoryConfigs_pulsarServiceUrl` is proxy URL(IP:port)
2. using helm to deploy pulsar
3. add a function
4. function pod is running in other namespace which I set in jobNamespace
function pod log shows
```
"Downloaded successfully"
shardId=0
[2020-06-11 08:29:11 +0000] [INFO] python_instance_main.py: Starting Python instance with Namespace(client_auth_params='file:///etc/auth/token', client_auth_plugin='org.apache.pulsar.client.impl.auth.AuthenticationToken', cluster_name='pulsar-145', dependency_repository=None, expected_healthcheck_interval=-1, extra_dependency_repository=None, function_details='{"tenant":"public","namespace":"default","name":"ken","className":"test_func.ExclamationFunction1","runtime":"PYTHON","autoAck":true,"parallelism":1,"source":{"inputSpecs":{"persistent://public/default/intput":{}},"cleanupSubscription":true},"sink":{"topic":"persistent://public/default/output"},"resources":{"cpu":1.1,"ram":"1073741824","disk":"10737418240"},"componentType":"FUNCTION"}', function_id='dd305aeb-a773-45d9-aa48-765c4fa451e5', function_version='fff11148-ed0d-4a65-b79f-8c6b7182ac88', hostname_verification_enabled='false', install_usercode_dependencies=True, instance_id='0', logging_config_file='/pulsar/conf/functions-logging/console_logging_config.ini', logging_directory='logs/functions', logging_file='ken', max_buffered_tuples='1024', metrics_port=9094, port=9093, pulsar_serviceurl='pulsar://<my-proxy-url>/', py='/pulsar/test_func.py', secrets_provider='secretsprovider.ClearTextSecretsProvider', secrets_provider_config=None, state_storage_serviceurl=None, tls_allow_insecure_connection='false', tls_trust_cert_path=None, use_tls='false')
2020-06-11 08:29:11.916 INFO Client:88 | Subscribing on Topic :persistent://public/default/intput
2020-06-11 08:29:11.916 INFO ConnectionPool:85 | Created connection for pulsar://<my-proxy-url>/
2020-06-11 08:29:11.918 INFO ClientConnection:330 | [10.244.166.160:35006 -> <my-proxy-url>] Connected to broker
2020-06-11 08:29:12.312 INFO HandlerBase:53 | [persistent://public/default/intput, public/default/ken, 0] Getting connection from pool
2020-06-11 08:29:12.350 INFO ConnectionPool:85 | Created connection for pulsar://pulsar-145-broker-0.pulsar-145-broker.pulsar.svc.cluster.local:6650
2020-06-11 08:29:12.351 INFO ClientConnection:332 | [10.244.166.160:35022 -> <my-proxy-url>] Connected to broker through proxy. Logical broker: pulsar://pulsar-145-broker-0.pulsar-145-broker.pulsar.svc.cluster.local:6650
2020-06-11 08:29:12.439 INFO ConsumerImpl:175 | [persistent://public/default/intput, public/default/ken, 0] Created consumer on broker [10.244.166.160:35022 -> <my-proxy-url>]
[2020-06-11 08:29:12 +0000] [INFO] server.py: Serving InstanceCommunication on port 9093
2020-06-11 08:39:12.313 INFO ConsumerStatsImpl:64 | Consumer [persistent://public/default/intput, public/default/ken, 0] , ConsumerStatsImpl (numBytesRecieved_ = 0, totalNumBytesRecieved_ = 0, receivedMsgMap_ = {}, ackedMsgMap_ = {}, totalReceivedMsgMap_ = {}, totalAckedMsgMap_ = {})
2020-06-11 08:49:12.313 INFO ConsumerStatsImpl:64 | Consumer [persistent://public/default/intput, public/default/ken, 0] , ConsumerStatsImpl (numBytesRecieved_ = 0, totalNumBytesRecieved_ = 0, receivedMsgMap_ = {}, ackedMsgMap_ = {}, totalReceivedMsgMap_ = {}, totalAckedMsgMap_ = {})
```
using `pulsar-admin topics stats public/default/intput` will return Topic not found
**Expected behavior**
function pod will subscript input topic and the topic will be created
**Desktop (please complete the following information):**
- OS: ubuntu
**Additional context**
pulsar-all:2.5.1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7273: Functions doesn't subscribe input topic
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7273:
URL: https://github.com/apache/pulsar/issues/7273#issuecomment-643866513
@btsken it seems that the consumers are already created. Did you use the same pulsar cluster?
```
2020-06-11 08:39:12.313 INFO ConsumerStatsImpl:64 | Consumer [persistent://public/default/intput, public/default/ken, 0] , ConsumerStatsImpl (numBytesRecieved_ = 0, totalNumBytesRecieved_ = 0, receivedMsgMap_ = {}, ackedMsgMap_ = {}, totalReceivedMsgMap_ = {}, totalAckedMsgMap_ = {})
2020-06-11 08:49:12.313 INFO ConsumerStatsImpl:64 | Consumer [persistent://public/default/intput, public/default/ken, 0] , ConsumerStatsImpl (numBytesRecieved_ = 0, totalNumBytesRecieved_ = 0, receivedMsgMap_ = {}, ackedMsgMap_ = {}, totalReceivedMsgMap_ = {}, totalAckedMsgMap_ = {})
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] btsken commented on issue #7273: Functions doesn't subscribe input topic
Posted by GitBox <gi...@apache.org>.
btsken commented on issue #7273:
URL: https://github.com/apache/pulsar/issues/7273#issuecomment-643885348
It's the same pulsar cluster
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] yjxyy commented on issue #7273: Functions doesn't subscribe input topic
Posted by GitBox <gi...@apache.org>.
yjxyy commented on issue #7273:
URL: https://github.com/apache/pulsar/issues/7273#issuecomment-653979626
I also encountered the same problem, how to solve it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] yjxyy commented on issue #7273: Functions doesn't subscribe input topic
Posted by GitBox <gi...@apache.org>.
yjxyy commented on issue #7273:
URL: https://github.com/apache/pulsar/issues/7273#issuecomment-653979878
ConsumerStatsImpl (numBytesRecieved_ = 0, totalNumBytesRecieved_ = 0, receivedMsgMap_ = {}, ackedMsgMap_ = {}, totalReceivedMsgMap_ = {}, totalAckedMsgMap_ = {})
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org