You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/11/08 09:11:02 UTC
Slack digest for #dev - 2019-11-08
2019-11-07 10:00:23 UTC - Anand Sinha: Can Bookkeeper Table Service to store state values ( <https://docs.google.com/document/d/155xAwWv5IdOitHh1NVMEwCMGgB28M3FyMiQSxEpjE-Y/edit#heading=h.56rbh52koe3f> ) be used *outside of* Pulsar Functions?
I would like to store and fetch state for keys from my key-shared consumer code and I was wondering if I could utilize the Table Service.
----
2019-11-07 10:07:29 UTC - Sijie Guo: technically yes.
you just need to create a table service client to talk to the table service, just like how functions interact with it.
----
2019-11-07 10:08:29 UTC - Anand Sinha: Thanks Sijie. Are there any samples anywhere? I would like to test it on a Standalone Pulsar instance.
----
2019-11-07 10:40:03 UTC - hangc: @hangc has joined the channel
----
2019-11-07 10:54:29 UTC - listar: @listar has joined the channel
----
2019-11-07 10:54:59 UTC - listar: anybody use pulsar in production envertionment ?
----
2019-11-07 10:55:10 UTC - listar: is there many bugs?
----
2019-11-07 10:55:23 UTC - listar: neeed to know thanks
----
2019-11-07 12:07:47 UTC - Anand Sinha: @Sijie Guo, any pointers would help.
----
2019-11-07 12:18:21 UTC - Sijie Guo: <https://github.com/apache/pulsar/blob/master/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/StateContextImpl.java>
you can check how functions use table
----
2019-11-07 12:34:01 UTC - hangc: we want to using pulsar to replace kafka in product environments, could anyone share some experiences or ideas ?
----
2019-11-07 12:43:07 UTC - Sijie Guo: There are many users moving from Kafka to Pulsar. It depends on your use case and how do you want to do the migration.
There are some stories you can read.
<https://medium.com/@yuvarajl/why-nutanix-beam-went-ahead-with-apache-pulsar-instead-of-apache-kafka-1415f592dbbb>
<https://streamnative.io/success-stories/>
----
2019-11-07 12:55:47 UTC - hangc: In my company, kafka producers are RDKafka、flume、flink、spark、kafka python client，and consumers are rdKafka、flume、flink、spark、kafka python client，we want to using pulsar replace kafka step by step, moving application one by one, but i am not sure whether where are some pits in pulsar when using pulsar, could you give me some ideas?
----
2019-11-07 19:51:52 UTC - Addison Higham: we use mostly Kinesis, so our experience is different, but we do use flink and lambda a lot. So far, the biggest gaps are mostly just in pulsar being newer and not having as many integrations, but we knew that going into it. From a technical perspective, it has been *fairly* issue free, we have found a few bugs, mostly in admin tooling. The core pulsar messaging has been really quite reliable and scaled to meet our needs so far
----
2019-11-07 22:21:35 UTC - Rafal Gorzkowski: @Rafal Gorzkowski has joined the channel
----
2019-11-08 07:51:39 UTC - Anand Sinha: Thanks, @Sijie Guo. I am assuming that we have to follow the code in `setupStateTable()` method in
<https://github.com/apache/pulsar/blob/master/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java>
to create the `Table` object. Is my understanding correct? Is there another direct way to create the Table instance?
----
2019-11-08 08:05:09 UTC - hangc: do the community have plan to release pulsar 2.5.0?
----
2019-11-08 08:05:39 UTC - Ali Ahmed: yes the current focus is on 2.4.2
----
2019-11-08 08:05:55 UTC - Ali Ahmed: 2.5.0 is not yet feature complete
----
2019-11-08 08:07:46 UTC - hangc: which version the pulsar-flink-connector feature will release?
----
2019-11-08 08:08:47 UTC - Ali Ahmed: there is already a flink connector <https://mvnrepository.com/artifact/org.apache.pulsar/pulsar-flink>
----
2019-11-08 08:09:09 UTC - Ali Ahmed: a improved one will arrive in 2.5.0
----
2019-11-08 08:11:07 UTC - hangc: cool
----
2019-11-08 08:40:23 UTC - Sijie Guo: @hangc :
- the 2.4.2 release is coming in 1~2 weeks.
- the 2.5.0 release still has a lot of features incomplete. the community is discussing to move to a time based release schedule. hope we can release 2.5.0 sooner.
- there is a pulsar-flink (with flink 1.6) connector in the pulsar repo.
- we developed a new pulsarr-flink (flink 1.9+) connector: <https://github.com/streamnative/pulsar-flink> (with exactly-once source, schema integration, flink sql integration, and etc). we are working with Flink committer on contributing this connector back to Flink mainstream. <https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector> (@yijie is driving the effort with the Flink community). It was planned for Flink 1.10. but waiting for FLIP-27 going in first.
----
2019-11-08 08:42:11 UTC - Sijie Guo: @Ali Ahmed we will contribute the newer connector to Flink main repo, not to Pulsar repo.
----
2019-11-08 08:53:45 UTC - hangc: cool ! I hope the good job!
----
2019-11-08 08:55:10 UTC - Sijie Guo: @hangc if you are using Flink 1.9+ and wanna to try pulsar-flink, I would suggest trying out <https://github.com/streamnative/pulsar-flink> . As it is the one that will be in Flink main repo.
----
2019-11-08 08:59:57 UTC - hangc: @Sijie Guo in my production environment, we mainly use Flink 1.7 & 1.9, thanks a lot!
----