You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/08/04 09:11:03 UTC
Slack digest for #general - 2018-08-04
2018-08-03 11:46:47 UTC - Jonny Daenen: @Jonny Daenen has joined the channel
----
2018-08-03 11:46:59 UTC - Tom Artoos: @Tom Artoos has joined the channel
----
2018-08-03 14:09:47 UTC - Josh West: <!here> for some reason i cannot list topics in my namespace using pulsar 2.1.0
----
2018-08-03 14:09:58 UTC - Josh West: ```# pulsar-admin topics list core-platform/testing
HTTP 500 Internal Server Error
Reason: Some error occourred on the server
```
----
2018-08-03 14:10:04 UTC - Josh West: i'm superuser too
----
2018-08-03 14:13:14 UTC - Josh West:
----
2018-08-03 14:24:25 UTC - Josh West: this feels like a bug with Pulsar 2.1.0 when using TLS authentication/authorization
----
2018-08-03 14:24:50 UTC - Josh West: i suspect the `pulsar-admin topics` command is calling an old API?
----
2018-08-03 14:26:24 UTC - Josh West: ``` curl -k --cert /tls/client.pem --key /tls/client-key.pem <https://pulsar-broker:8443/admin/v2/persistent/core-platform/testing/> | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 49 100 49 0 0 1115 0 --:--:-- --:--:-- --:--:-- 1139
[
"<persistent://core-platform/testing/pulsarperf>"
]
```
----
2018-08-03 14:26:35 UTC - Josh West: ```# curl -k --cert /tls/client.pem --key /tls/client-key.pem <https://pulsar-broker:8443/admin/v2/non-persistent/core-platform/testing/>
--- An unexpected error occurred in the server ---
Message: HTTP 401 Authentication required
Stacktrace:
org.apache.pulsar.client.admin.PulsarAdminException$NotAuthorizedException: HTTP 401 Authentication required
at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:170)
at org.apache.pulsar.client.admin.internal.TopicsImpl$5.failed(TopicsImpl.java:161)
at org.glassfish.jersey.client.JerseyInvocation$7.failed(JerseyInvocation.java:965)
at org.glassfish.jersey.client.JerseyInvocation$7.completed(JerseyInvocation.java:952)
at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:196)
at org.glassfish.jersey.client.ClientRuntime.access$300(ClientRuntime.java:74)
at org.glassfish.jersey.client.ClientRuntime$2$1$1.run(ClientRuntime.java:166)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340)
at org.glassfish.jersey.client.ClientRuntime$2$1.response(ClientRuntime.java:164)
at org.glassfish.jersey.client.internal.HttpUrlConnector$3.run(HttpUrlConnector.java:297)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at jersey.repackaged.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:50)
at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:37)
at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:293)
at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:180)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340)
at org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:208)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.ws.rs.NotAuthorizedException: HTTP 401 Authentication required
at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1014)
at org.glassfish.jersey.client.JerseyInvocation.access$900(JerseyInvocation.java:92)
... 32 more
```
----
2018-08-03 14:33:19 UTC - Ivan Kelly: @Josh West are you using the proxy or going straight to broker?
----
2018-08-03 14:39:00 UTC - Josh West: straight to the broker
----
2018-08-03 14:39:20 UTC - Josh West: my understanding from issues/docs @Ivan Kelly is that the pulsar proxy doesn't work with TLS authentication
----
2018-08-03 14:39:33 UTC - Ivan Kelly: it does from 2.1
----
2018-08-03 14:39:39 UTC - Josh West: ooohh nice
----
2018-08-03 14:41:21 UTC - Ivan Kelly: could you post your broker config?
----
2018-08-03 18:19:43 UTC - Josh West: hmmmmmmmmmmmmmmmmmmm
----
2018-08-03 18:19:49 UTC - Josh West: are roles not allowed to have forward slashes?
----
2018-08-03 18:19:51 UTC - Josh West: in their name
----
2018-08-03 18:22:19 UTC - Josh West: ```root@pulsar-admin:/pulsar# pulsar-admin namespaces grant-permission --actions produce,consume --role '<spiffe://developer/passport>' core-platform/testing
HTTP 404 Not Found
Reason: HTTP 404 Not Found
root@pulsar-admin:/pulsar# pulsar-admin namespaces grant-permission --actions produce,consume --role 'spiffe:\/\/developer\/passport' core-platform/testing
HTTP 404 Not Found
Reason: HTTP 404 Not Found
root@pulsar-admin:/pulsar# pulsar-admin namespaces grant-permission --actions produce,consume --role 'foo' core-platform/testing
root@pulsar-admin:/pulsar# pulsar-admin namespaces permissions core-platform/testing
{
"foo" : [ "produce", "consume" ]
}```
----
2018-08-03 18:26:40 UTC - Josh West: @Sijie Guo this looks like a `pulsar-admin` issue as well with URL encoding
----
2018-08-03 18:26:55 UTC - Josh West: if i write out the role using %2F instead of / then the command succeeds
----
2018-08-03 18:27:17 UTC - Josh West: ```root@pulsar-admin:/pulsar# pulsar-admin namespaces grant-permission --actions produce,consume --role 'spiffe:%2F%2Fdeveloper%2Fpassport' core-platform/testing
root@pulsar-admin:/pulsar# curl -k --cert /tls/client.pem --key /tls/client-key.pem <https://pulsar-broker:8443/admin/v2/namespaces/core-platform/testing/permissions> | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 81 100 81 0 0 1819 0 --:--:-- --:--:-- --:--:-- 1840
{
"foo": [
"consume",
"produce"
],
"<spiffe://developer/passport>": [
"consume",
"produce"
]
}
```
----
2018-08-03 18:32:28 UTC - Sijie Guo: <!here> Apache Pulsar 2.1.0-incubating has been released!
It includes following major features:
1) Pulsar IO: a connector framework for moving data in and out of Pulsar using Pulsar Functions. 6 builtin connectors are included. <https://pulsar.incubator.apache.org/docs/en/io-overview/>
2) Tiered Storage to offload old segments to long term storage, such as S3, HDFS. S3 is currently supported in this release. <https://pulsar.incubator.apache.org/docs/en/concepts-tiered-storage/>
3) Pulsar Functions introduces a new State API to store function state back to BookKeeper. It is available as a developer-preview feature in Java SDK.
<https://pulsar.incubator.apache.org/docs/en/functions-state/>
4) A c++ wrapped go client: <https://pulsar.incubator.apache.org/docs/en/client-libraries-go/>
5) More Schemas are supported, including Avro and Protobuf.
Download: <http://pulsar.apache.org/download/>
Release Notes: <http://pulsar.apache.org/release-notes/#2.1.0-incubating>
Docker image: <https://hub.docker.com/r/apachepulsar/pulsar/>
- `apachepulsar/pulsar:2.1.0-incubating` : a basic image include server components only.
- `apachepulsar/pulsar-all:2.1.0-incubating`: a include-everything image, including all built-in connectors.
C++ Client :
- RPMS: <http://pulsar.apache.org/docs/en/client-libraries-cpp/#rpm>
- DEBs : <http://pulsar.apache.org/docs/en/client-libraries-cpp/#deb>
Python Client: <https://pypi.org/project/pulsar-client/2.1.0/>
bananadance : Josh West, Sanjeev Kulkarni, Ali Ahmed, jia zhai
heart_eyes : Poule, Ali Ahmed, jia zhai
clap : Grant Wu, Ali Ahmed, Adam Williams, Poule, Matteo Merli, jia zhai, Karthik Ramasamy
unicorn_face : Poule, Adam Williams, Ali Ahmed, jia zhai, Karthik Ramasamy
----
2018-08-03 18:33:27 UTC - Sijie Guo: @Josh West: sorry was typing the release message, let me check your question now
----
2018-08-03 18:36:46 UTC - Sijie Guo: yeah I see. I think pulsar internally uses URL encoding for roles, cluster-names. so there is probably somewhere doesn’t handle URL encoding correctly. let me fire an issue.
----
2018-08-03 18:39:20 UTC - Sijie Guo: I created <https://github.com/apache/incubator-pulsar/issues/2304>
----
2018-08-03 18:39:32 UTC - Sijie Guo: will be looking into a fix today or monday
----
2018-08-03 18:52:54 UTC - Poule: nice work
----
2018-08-03 18:54:27 UTC - Sijie Guo: @Sijie Guo set the channel topic: General channel - Pulsar release 2.1.0-incubating - <https://pulsar.incubator.apache.org/release-notes/#2.1.0-incubating>
----
2018-08-03 18:58:32 UTC - Poule: great that schemas are in bookkeeper, and the State for functions is great too
blush : Sijie Guo, Karthik Ramasamy
----
2018-08-03 19:00:50 UTC - Poule: great to have the GetState functionality
----
2018-08-03 19:37:52 UTC - Knkn: @Knkn has joined the channel
----
2018-08-03 19:45:34 UTC - Facundo Glaeser: @Facundo Glaeser has joined the channel
----
2018-08-03 19:59:50 UTC - Facundo Glaeser: Hi, I am new to Apache Pulsar. I currently evaluating pulsar to replace our messaging system (IBM MQ). I would like to know if there a way to subscribe to multiples topics using the websocket api? As a workaround I'm currently using pulsar functions to route multiples topics to a single topic.
----
2018-08-03 20:04:46 UTC - Sijie Guo: @Facundo Glaeser currently websocket doesn’t support multi-topic consumers. however just be curious, can’t you subscribe the topics one by one and using one message listener?
----
2018-08-03 20:18:16 UTC - Facundo Glaeser: @Sijie Guo I've never tried that, I don't know if it is possible. Do you think is possible?
----
2018-08-03 20:19:37 UTC - Facundo Glaeser: Hi all, is there someone working in a pulsar netcore client?:grinning:
----
2018-08-03 20:20:39 UTC - Grant Wu: I mean, it’s just using one connection per subscription
----
2018-08-03 20:22:02 UTC - Facundo Glaeser: @Grant Wu I agree.
----
2018-08-03 20:26:41 UTC - Sijie Guo: @Facundo Glaeser we are catching up features on non-java clients. cpp/python are almost caught up. we will do more in go and websocket in the coming release. so we will try to do that multi-topics. for now, if you probably can get around with one connection per subscription
----
2018-08-03 20:29:32 UTC - Sijie Guo: @Facundo Glaeser is netcore .NET?
currently I am not aware of any work on .net client. if there is a demand on .net, we probably prioritize that. it would be simple to wrap over existing c++ client. or if you are interested in contribution, that would be great as well
----
2018-08-03 20:33:57 UTC - Facundo Glaeser: netcore is cross platform version of .Net Framework. We are currently migrating our apps to a microservices arq. using netcore and docker.
----
2018-08-03 22:11:00 UTC - Facundo Glaeser: @Sijie Guo I think that I can start build some netcore client. Please correct me if I wrong. As far as know the java client is built using netty, I've been doing some research and there is a netcore port of netty called DotNetty, I think I can start from that.
----
2018-08-03 22:13:17 UTC - Grant Wu: Are you planning on building on top of the WebSockets API or the protobuf protocol?
----
2018-08-03 22:14:44 UTC - Sijie Guo: @Facundo Glaeser
the is one way to do that. although I am not sure that is the best route.
an alternative is following what other language clients (python, cgo) are doing, create a netcore client using the c++ client library (although I am not sure if netcore is able to do that or not).
----
2018-08-03 22:15:05 UTC - Sijie Guo: @Grant Wu I think he means protobuf protocol :slightly_smiling_face:
----
2018-08-03 22:15:54 UTC - Facundo Glaeser: @Grant Wu the protobuf protocol
----
2018-08-03 22:16:52 UTC - Grant Wu: <https://github.com/dotnet/corefx/issues/24444> maybe?
----
2018-08-03 22:18:31 UTC - Facundo Glaeser: @Grant Wu Thanks, I will check that.
----
2018-08-03 22:38:04 UTC - Facundo Glaeser: @Grant Wu The only "problem" I see, is that the c++ lib is just for linux and MacOS and not for Windows. I have a lot of application running on windows right now.
----
2018-08-03 22:41:47 UTC - Ali Ahmed: why would c++ only be for macos and linux ?
----
2018-08-03 22:44:54 UTC - Sijie Guo: @Facundo Glaeser: We only document macos and linux because most of the people are kind of more familiar with these two platforms. But I think you should be able to build a windows c++ client (although I haven’t really tried that before).
----
2018-08-03 22:47:44 UTC - Facundo Glaeser: I said "problem" for that, the doc says that was only tested in Mac and linux only.
----
2018-08-03 22:50:06 UTC - Sijie Guo: :slightly_smiling_face:
----
2018-08-03 22:50:34 UTC - Ali Ahmed: it all depends on community demand, if more people are requesting .net client it will be prioritized more
----
2018-08-03 22:53:31 UTC - Facundo Glaeser: @Ali Ahmed Where I can request that? :grin:
----
2018-08-03 22:54:06 UTC - Ali Ahmed: dev mailing list and or <https://github.com/apache/incubator-pulsar/issues>
----
2018-08-03 22:54:32 UTC - Facundo Glaeser: :+1:
----
2018-08-03 23:17:57 UTC - Poule: I request Python client to be on par with Java client before doing .NET :innocent:
----
2018-08-03 23:18:42 UTC - Poule: `State currently is not supported at Python SDK.`
`The Pulsar schema registry is currently available only for the Java client.`
----
2018-08-03 23:20:43 UTC - Sijie Guo: @Poule :slightly_smiling_face: we are catching up at c++ and python. you can checkout - <https://github.com/apache/incubator-pulsar/wiki/Client-Features-Matrix>
----
2018-08-03 23:21:03 UTC - Sijie Guo: I should probably put a reference in the webpage.
----
2018-08-03 23:30:53 UTC - Poule: where is State API in the feature matrix?
----
2018-08-03 23:31:57 UTC - Sijie Guo: oh we don’t have feature matrix for functions yet. that is for clients. will create one for functions
----