You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/07/27 09:11:02 UTC
Slack digest for #general - 2018-07-27
2018-07-26 09:11:39 UTC - Ali Ahmed: I wouldn’t recommend kstreams since the state is local and can be lost
----
2018-07-26 09:13:05 UTC - Mate Varga: > durable state functions
thanks -- any doc about this?
----
2018-07-26 09:13:40 UTC - Mate Varga: I thought kstreams can just be a view of the state stored on a topic, which'd be enough (i.e. it could be reinitialized at startup)
----
2018-07-26 09:14:06 UTC - Ali Ahmed: here are WIP docs <https://pulsar.incubator.apache.org/docs/latest/functions/overview/>
----
2018-07-26 09:15:26 UTC - Ali Ahmed: @Tomer Lev did you originally setup two clusters ?
----
2018-07-26 09:15:52 UTC - Tomer Lev: no
----
2018-07-26 09:15:57 UTC - Ali Ahmed: @Mate Varga by default kstreams uses rocksDB
----
2018-07-26 09:16:46 UTC - Ali Ahmed: it can be replicated to kafka and log compacted, but I don’t think there are any strong durability guarantees.
----
2018-07-26 09:16:58 UTC - Mate Varga: Thanks.
----
2018-07-26 09:17:25 UTC - Mate Varga: Would you recommend using Pulsar Functions if we need to roll out something simple to production in ~2 months?
----
2018-07-26 09:17:38 UTC - Mate Varga: (no latency requirements, but it should 'mostly work' :slightly_smiling_face: )
----
2018-07-26 09:17:54 UTC - Ali Ahmed: @Tomer Lev I really don’t know , did you use some scripts to deploy ?
----
2018-07-26 09:18:26 UTC - Tomer Lev: No, the only thing is I might initialized twice or something
----
2018-07-26 09:18:39 UTC - Tomer Lev: after changing configs maybe
----
2018-07-26 09:18:51 UTC - Tomer Lev: that's the only thing in my mind
----
2018-07-26 09:19:18 UTC - Ali Ahmed: @Mate Varga yes I can recommend it for usage , the programming model is very simple and it’s stable enough for production usage
----
2018-07-26 09:19:48 UTC - Ali Ahmed: @Tomer Lev I don’t think running the script twice can occur zookeeper won’t allow it
----
2018-07-26 09:20:44 UTC - Ali Ahmed: unless you have initialized two different cluster id’s
----
2018-07-26 09:21:40 UTC - Ali Ahmed: will need logs to understand what’s happening
----
2018-07-26 09:24:09 UTC - Mate Varga: I have difficulties finding specific info about state storage in the docs (about functions). Should I just RTFC?
----
2018-07-26 09:24:43 UTC - Mate Varga: (Is that stored in the 'Context'?)
----
2018-07-26 09:28:12 UTC - Sijie Guo: @Mate Varga the stateful functions will be released as developer preview as part of upcoming 2.1.0 release. the documentation hasn’t been fully updated for 2.1.0 release.
for now, you can get a quick impression by checking out this word count function example: <https://github.com/apache/incubator-pulsar/blob/master/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/WordCountFunction.java>
or you can checkout the instructions that we are using for validating 2.1.0 release : <https://github.com/apache/incubator-pulsar/wiki/Release-Candidate-Validation#validate-stateful-functions>
(hope this helps)
----
2018-07-26 09:29:13 UTC - Mate Varga: TY!
----
2018-07-26 09:31:02 UTC - Sijie Guo: @Tomer Lev can you share a screenshot of your dashboard? or maybe describe the sequence on how did you initialize twice?
----
2018-07-26 15:38:05 UTC - Charles Hutchinson: Trying to get 2.0.1 setup. Zookeeper fails to start with an invalide config when I have the zkservers in the zookeeper.conf file as:
```server.1=zkserver1.fqdn
server.2=zkserver2.fqdn
server.3=zkserver3.fqdn
server.4=zkserver4.fqdn
server.5=zkserver5.fqdn```
----
2018-07-26 15:38:48 UTC - Charles Hutchinson: If I comment out the servers in the config file `bin/pulsar-daemon start zookeeper` works.
----
2018-07-26 15:39:17 UTC - Charles Hutchinson: Do I not need to call out all of the servers in the local zk cluster with this version of pulsar?
----
2018-07-26 15:47:07 UTC - Charles Hutchinson: sorry, those lines are `server.1=zkserver1.fqdn:2888:3888`
----
2018-07-26 15:50:59 UTC - Charles Hutchinson:
----
2018-07-26 17:09:56 UTC - Matteo Merli: @Charles Hutchinson if you remove the servers lists, basically ZK starts in standalone mode.
Have you set the `myid` file?
Based on <http://pulsar.apache.org/docs/latest/deployment/cluster/#DeployingaZooKeepercluster-flqel9>
Each ZK server needs to have a file with the “id” to know which server they are within the list. Eg:
On server 1:
```
echo 1 > data/zookeeper/myid
```
on server 2:
```
echo 2 > data/zookeeper/myid
```
This is typically automated in some form. For example with Ansible: <https://github.com/apache/incubator-pulsar/blob/master/deployment/terraform-ansible/deploy-pulsar.yaml#L81>
The file is filled up from a template
----
2018-07-26 17:12:15 UTC - Charles Hutchinson: Yes, all servers have the myid file populated to match the server.# that is defined in zookeeper.conf
----
2018-07-26 17:23:33 UTC - Martin Kunev: Hello, I have an issue when I try to create a tenant in pulsar 2 using the REST API. If I understand correctly I need to send a `PUT` request to `http://<host>/admin/v2/tenants/<tenant_name>` and specify parameters in the body of the request: `{"adminRoles": ["admin"], "allowedClusters": ["cluster0","cluster1"]}`. However I am getting error `415 Unsupported Media Type` when I execute this request. When I leave the body empty, I get no error. Am I doing something wrong?
----
2018-07-26 17:27:59 UTC - Matteo Merli: You need to set content type to json in the request. eg: `curl -H'Content-Type: application/json' ..`
----
2018-07-26 17:28:59 UTC - Matteo Merli: Ok, can you paste the ZK exception? Is there anything telling there?
----
2018-07-26 17:32:09 UTC - Charles Hutchinson: When I have the servers in the zookeeper.conf nothing gets written to the pulsar*.log. I pasted the contents of the pulsar-*.out file that does get written to.
----
2018-07-26 17:38:48 UTC - Matteo Merli: uhm, that’s strange, could it some extra characters?
Can you post the zookeeper.conf in a gist or DM ?
----
2018-07-26 18:05:35 UTC - Martin Kunev: Thank you very much @Matteo Merli - this fixed the error.
----
2018-07-26 18:12:03 UTC - Surya Prakash Munigoti: @Surya Prakash Munigoti has joined the channel
----
2018-07-26 18:12:18 UTC - Martin Kunev: However I now see a different error. Once I have the tenant, I try to create a namespace (`PUT /admin/v2/namespaces/<tenant_name>/<namespace_name>`), but get 404. Further investigation shows that pulsar believes the tenant doesn't exist:
```$ bin/pulsar-admin tenants list
public
mytenant
$ bin/pulsar-admin namespaces list mytenant
HTTP 404 Not Found
Reason: Tenant does not exist
```
----
2018-07-26 19:07:16 UTC - Daniel Ferreira Jorge: @Martin Kunev You first need to create the tenant with `PUT /admin/v2/tenants/<tenant_name>` with `application/json` body parameters with a list of `adminRoles` and a list of `allowedClusters`... something like `{"adminRoles": ["admin"], "allowedClusters": ["standalone"]}`
----
2018-07-26 19:11:49 UTC - Daniel Ferreira Jorge: @Martin Kunev The body parameters of the REST API are not documented so for now, you have to figure how to use the REST API by digging into the source code but I believe there is already an issue open to correct this problem: <https://github.com/apache/incubator-pulsar/issues/1650>
----
2018-07-26 19:14:55 UTC - Martin Kunev: @Daniel Ferreira Jorge I actually get this error after having successfully created the tenant while setting adminRoles and allowedClusters. As I mentioned in my previous comment, the tenant also appears in the output of `bin/pulsar-admin tenants list`.
----
2018-07-26 19:15:48 UTC - Matteo Merli: If you try to do:
`bin/pulsar-admin tenants get mytenant`. Does that work?
----
2018-07-26 19:16:23 UTC - Daniel Ferreira Jorge: @Martin Kunev did you create the namespace with `{"replication_clusters": ["standalone"]}`? There is also this caveat
----
2018-07-26 19:19:01 UTC - Martin Kunev: The code I'm using creates the namespace first and then sets replication clusters. Up until now this has worked. I actually didn't know I can set the replication clusters during namespace creation. I'll try that.
----
2018-07-26 19:21:31 UTC - Matteo Merli: Yes, you can pass directly the namespace configuration as part of the JSON body when creating the namespace. In any, the list namespaces should work in either case…
----
2018-07-26 20:13:00 UTC - Josh West: what's the official way to build the Docker containers for Pulsar -- ie building containers for the Pulsar broker/zookeeper/bookeeper/etc from github
----
2018-07-26 20:13:38 UTC - Josh West: `docker/build.sh` ?
----
2018-07-26 20:14:12 UTC - Sijie Guo: mvn clean install -DskipTests -Pdocker
----
2018-07-26 20:14:49 UTC - Sijie Guo: @Josh West use `-Pdocker` when you are building pulsar project, it will activate docker build profile to build docker images
----
2018-07-26 20:15:13 UTC - Sijie Guo: docker/build.sh requires you build the pulsar project first to produce binary tgz
----
2018-07-26 20:15:59 UTC - Josh West: nice @Sijie Guo thanks; trying now
----
2018-07-26 20:16:13 UTC - Josh West: watches his laptop hover over the desk as the fan spins on high
ghost : Sijie Guo
helicopter : Grant Wu
----
2018-07-26 20:17:37 UTC - Josh West: ```[Thu Jul 26 16:15:39 2018] CPU0: Core temperature above threshold, cpu clock throttled (total events = 47)
[Thu Jul 26 16:15:39 2018] CPU3: Package temperature above threshold, cpu clock throttled (total events = 47)
[Thu Jul 26 16:15:39 2018] CPU2: Core temperature above threshold, cpu clock throttled (total events = 47)
[Thu Jul 26 16:15:39 2018] CPU1: Package temperature above threshold, cpu clock throttled (total events = 47)
[Thu Jul 26 16:15:39 2018] CPU2: Package temperature above threshold, cpu clock throttled (total events = 47)
[Thu Jul 26 16:15:39 2018] CPU0: Package temperature above threshold, cpu clock throttled (total events = 47)
```
fire : Grant Wu
fire_engine : Grant Wu
rotating_light : Grant Wu
shower : Grant Wu
computer : Grant Wu
put_litter_in_its_place : Grant Wu
----
2018-07-26 20:17:38 UTC - Josh West: heh
----
2018-07-26 20:28:24 UTC - Josh West: hahaha @Grant Wu
wink : Grant Wu
----
2018-07-26 20:29:05 UTC - Grant Wu: my personal laptop honestly has this 24/7 just from Chrome
ghost : Sijie Guo
----