You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/11/05 09:11:02 UTC
Slack digest for #general - 2019-11-05
2019-11-04 09:40:10 UTC - Jasper Li: Halo all,
I have a question to use Pulsar SQL after doing cdc by debezium connector, it returns me an error as below:
```
2019-11-04T09:32:03.590Z WARN statement-response-2 com.facebook.presto.server.ThrowableMapper Request failed for /v1/statement/20191104_093201_00002_562v2/3
java.lang.IllegalArgumentException: Unsupported schema : {
"name": "db.db.table",
"schema": {
"key": {
"name": "Bytes",
"schema": "",
"type": "BYTES",
"properties": {}
},
"value": {
"name": "Bytes",
"schema": "",
"type": "BYTES",
"properties": {}
}
},
"properties": {
"key.schema.name": "Bytes",
"key.schema.properties": "{}",
"key.schema.type": "BYTES",
"kv.encoding.type": "INLINE",
"value.schema.name": "Bytes",
"value.schema.properties": "{}",
"value.schema.type": "BYTES"
}
}
```
Does it mean there is no way to query data from debezium cdc record by Pulsar SQL, unless I have transform it?
Thanks!!!!!
----
2019-11-04 09:48:12 UTC - Jasper Li: @xiaolong.ran Thank you very much!!! I am successful in offloading to my GCS bucket now!!! It is cool!!!
----
2019-11-04 10:09:52 UTC - Jasper Li: Halo again,
I have another question about debezium CDC in Pulsar compare with Kafka. Since I am moving from Kafka to Pulsar, I have used debezium to do cdc and get change log from mysql, but, I have used ```transforms.unwrap.type=io.debezium.transforms.UnwrapFromEnvelope``` and ```key.converter=io.confluent.connect.avro.AvroConverter/value.converter=io.confluent.connect.avro.AvroConverter``` in Kafka connect, but I cannot use them in Pulsar IO directly, is it possible to apply them in Pulsar IO?
Thanks again!!!
----
2019-11-04 10:56:12 UTC - Kabeer Ahmed: @tuteng Thank you for tagging @yijie
----
2019-11-04 11:10:32 UTC - Sijie Guo: @Jasper Li:
for your first question, I think PrestoSQL doesn’t support key/value schema yet. @Penghui Li was looking into adding that support.
for the second question, I believe you can set those settings. you can just add those debezium settings under `configs:` section in the yaml file you used to submit the connector.
+1 : Jasper Li, Penghui Li
----
2019-11-04 13:09:35 UTC - Berger: @Berger has joined the channel
----
2019-11-04 13:17:26 UTC - Berger: Good morning everyone. I’m looking for message solution which can be installed as multicluster between two different cloud providers at the start and add more clusters installed between different infrastructures across world :slightly_smiling_face:
I started with basic installation on k8s cluster, but here I start to think if it is possible to install pulsar on multiple k8s cluster and connect them together into one multi-cluster instance. Is it something what I can gain with the kubernetes, or I have to use normal instances, but still there i need probably any quorum to share configuration between clusters :open_mouth:
+1 : Jasper Li
----
2019-11-04 13:57:10 UTC - Matt Mitchell: I tried to run the latest dashboard (last week and tip of master as of 5 mins ago) and it failed with `sudo: initdb: command not found`. I found that the path to `initdb` in the `init-postgres.sh` script references `9.6` but the image has `11`. After updating the script, the dashboard seems to work. Is this a known issue or possibly something related to my env?
----
2019-11-04 14:08:56 UTC - Matt Mitchell: Also, once it’s running, I the UI doesn’t list anything… no tenants, namespaces etc.. maybe there’s a config/ENV option missing from the README?
----
2019-11-04 14:15:07 UTC - tuteng: You can try login in pg for query data?
----
2019-11-04 14:55:26 UTC - Sijie Guo: @Matt Mitchell the current dashboard uses topic stats for displaying tenants and namespaces. if you don’t have any traffic, those information might not show up. You can try use the new management console: <https://github.com/apache/pulsar-manager>
----
2019-11-04 14:57:12 UTC - Sijie Guo: you can install pulsar in multiple k8s clusters and expose the proxies through a load balancer. so each cluster can connect to the others. In this way, you can make a global instance.
----
2019-11-04 15:10:58 UTC - Berger: @Sijie Guo Did you seen any examples (articles or whatever) with such configuration? I guess in this case I can use this method to make connection between different installation types like k8s, normal installation on instances etc.
----
2019-11-04 15:18:38 UTC - Matt Mitchell: will do. thanks @Sijie Guo
----
2019-11-04 15:45:00 UTC - Alexandre DUVAL: @Jerry Peng do you have an example of function config yaml file?
----
2019-11-04 15:46:15 UTC - Alexandre DUVAL: Hi, how to inject env var to a pulsar function?
----
2019-11-04 15:50:34 UTC - Alex Rufo: @Alex Rufo has joined the channel
----
2019-11-04 15:53:20 UTC - Alexandre DUVAL: env vars should be passed in PULSAR_EXTRA_OPTS?
----
2019-11-04 16:03:16 UTC - Matteo Merli: @Jared Mackey @Raman Gupta the sequence id, a part for deduplication, is used to correlate a SendReceipt to a particular Send request.
It’s not optional or ignored, but rather it’s stored within the message metadata.
The sequence id is a per-producer client assigned identifier while the “message” is a storage assigned unique identifier
----
2019-11-04 18:55:21 UTC - Addison Higham: reading the docs on retention policies, want to make sure I understand something. The docs say "and" for size and time, does that mean that if I set a policy for 10GB size and 3 hours of time, that it could go beyond 10GB if I have more than 10GB of data in the 3 hour window?
----
2019-11-04 19:27:45 UTC - Jerry Peng: No it’s which limit is reached first
----
2019-11-04 19:45:06 UTC - Addison Higham: okay, that is what I thought was more likely the case (and maybe I missed it) but it wasn't obvious at first glance
----
2019-11-04 20:31:36 UTC - Jerry Peng: ```
name: jerry-function
tenant: public
namespace: default
jar: /Users/jerrypeng/workspace/incubator-pulsar/pulsar-functions/java-examples/target/pulsar-functions-api-examples.jar
className: org.apache.pulsar.functions.api.examples.TestFunction
inputSpecs:
<persistent://jerry/default/jerry-input>:
receiverQueueSize: 1000
output: <persistent://jerry/default/jerry-output>
parallelism: 1
cleanupSubscription: true
```
----
2019-11-04 21:17:20 UTC - Alexandre DUVAL: there is a way to define custom input schema?
----
2019-11-04 22:47:02 UTC - CTRL: @CTRL has joined the channel
----
2019-11-04 22:47:42 UTC - CTRL: hi everyone! :slightly_smiling_face:
wave : Chris Bartholomew, Matteo Merli, Karthik Ramasamy
----
2019-11-04 23:49:08 UTC - JJ: @JJ has joined the channel
----
2019-11-05 01:28:18 UTC - Jasper Li: Thanks for your reply!!! It is happy to know the PrestoSQL will support key/value schema in the future and I will try to set up my debezium again. :slightly_smiling_face:
----
2019-11-05 01:46:41 UTC - kay pan: @kay pan has joined the channel
----
2019-11-05 01:53:21 UTC - kay pan: hi everyone
----
2019-11-05 01:54:34 UTC - kay pan: i have a issue: [pulsar-ordered-OrderedExecutor-7-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x20043268f0f000b
----
2019-11-05 01:54:45 UTC - kay pan: please help ,thanks
----
2019-11-05 03:26:37 UTC - mrigesh: @mrigesh has joined the channel
----
2019-11-05 05:35:43 UTC - Gopi Krishna: <https://github.com/PharosProduction/tutorial-pulsar-java> so if we are writing java classes as in this link, how do we run the java classes of producers and consumers. I am confused
----
2019-11-05 06:52:01 UTC - Gopi Krishna: Are there any connectors by which we can stream data from mongodb to pulsar? I can find <https://pulsar.apache.org/docs/en/next/io-mongo-sink/> pulsar sink connector but not any connector to pull data from mongo
----
2019-11-05 06:52:27 UTC - Gopi Krishna: Are there any connectors by which we can stream data from mongodb to pulsar? I can find <https://pulsar.apache.org/docs/en/next/io-mongo-sink/> pulsar sink connector but not any connector to pull data from mongo
----
2019-11-05 06:52:48 UTC - Ali Ahmed: @Gopi Krishna You mean a cdc for mongodb ?
----
2019-11-05 06:53:11 UTC - Gopi Krishna: what is a cdc ?
----
2019-11-05 06:53:52 UTC - Ali Ahmed: change data capture
----
2019-11-05 06:55:30 UTC - Gopi Krishna: Hmm, not exactly. Basically I am trying to read the data streamed into mongodb through nifi. This data can be historical or real-time
----
2019-11-05 06:58:09 UTC - Gopi Krishna: any idea ?
----
2019-11-05 07:16:30 UTC - tuteng: You can try <https://pulsar.apache.org/docs/en/next/io-debug/> to debug
----
2019-11-05 07:16:50 UTC - tuteng: <https://pulsar.apache.org/docs/en/next/io-debug/#debug-in-localrun-mode>
----
2019-11-05 07:19:54 UTC - Gopi Krishna: This is just for debugging of mongo-connector-sink
----
2019-11-05 07:23:58 UTC - tuteng: <https://github.com/apache/pulsar/issues/5474> We haven't added mongo's cdc scene yet.
----
2019-11-05 07:24:14 UTC - tuteng: pull data from mongo
----
2019-11-05 07:24:55 UTC - Gopi Krishna: thanks will go through
----
2019-11-05 07:28:58 UTC - Sijie Guo: @tuteng: @Gopi Krishna is asking for a mongodb cdc.
----