You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/12/20 09:11:02 UTC
Slack digest for #general - 2018-12-20
2018-12-19 11:58:37 UTC - Samuel Sun: Hi, one question for schema , currently golang client support it or not ? thanks
----
2018-12-19 11:59:34 UTC - Sijie Guo: it is not supported yet. we are targeting at supporting it for 2.3.0 release.
----
2018-12-19 12:09:50 UTC - Ganga Lakshmanasamy: We would like to read data from a file and stream it into ES. The file can be from any remote server with proper access being given. Do we have any plugin already available to handle such condition?
----
2018-12-19 12:14:23 UTC - Samuel Sun: thanks
----
2018-12-19 12:17:56 UTC - Samuel Sun: another question, is it possible to get into the zk while using standalone mode ? Thanks : )
----
2018-12-19 12:21:20 UTC - Samuel Sun: seems like the result is empty:
----
2018-12-19 12:21:27 UTC - Samuel Sun: ./pulsar-managed-ledger-admin print-managed-ledger --zkServer 127.0.0.1 --managedLedgerPath /
----
2018-12-19 12:21:45 UTC - Sijie Guo: bin/pulsar zookeeper-shell -server localhost:2181
----
2018-12-19 12:22:07 UTC - Sijie Guo: you can use `bin/pulsar zookeeper-shell`
----
2018-12-19 12:24:02 UTC - Samuel Sun: nice, but managed-ledger-admin not work ?
----
2018-12-19 12:40:19 UTC - Sijie Guo: managed-ledger-admin also works
----
2018-12-19 12:42:01 UTC - Samuel Sun: managedLedgerPath should be ?
----
2018-12-19 12:42:21 UTC - Samuel Sun: ./bin/pulsar-managed-ledger-admin print-managed-ledger --zkServer localhost:2181 --managedLedgerPath / no results
----
2018-12-19 12:50:05 UTC - Sijie Guo: managed ledger path should be `<tenant>/<namespace>/[persistent|non-persistent]/<topic>`
----
2018-12-19 12:50:16 UTC - Sijie Guo: e.g. ’public/default/persistent/test-topic`
----
2018-12-19 12:51:41 UTC - Samuel Sun: got it, it works, thanks.
----
2018-12-19 12:54:30 UTC - Sijie Guo: :+1:
----
2018-12-19 13:57:08 UTC - Chris Miller: I'm curious to know the reason why message TTL, retention and backlog settings are applied at the namespace level. To me it seems more useful to have them at the topic level (since some topics I might want to keep forever, some are transient, some buffer for an hour etc). As it stands it seems I need to create a namespace for each specific set of policies, and there's no obvious way to change the settings for a topic other than move it to another namespace. Am I thinking about this the wrong way?
----
2018-12-19 13:58:14 UTC - Sijie Guo: @Chris Miller there is an outstanding task for adding policies at topic-level
----
2018-12-19 13:58:31 UTC - Chris Miller: Ah ok thanks, I guess it's not just me then
----
2018-12-19 13:59:19 UTC - Chris Miller: Found it, thanks for the pointer: <https://github.com/apache/pulsar/issues/2688>
----
2018-12-19 14:00:16 UTC - Sijie Guo: you are welcome
----
2018-12-19 14:45:02 UTC - Ryan Samo: @Sijie Guo no problem at all. I am currently out on holiday until next year but when I return to work I will try to put together a post. Thanks!
----
2018-12-19 16:30:04 UTC - David Kjerrumgaard: @Victor Li Try both of these NAR files
----
2018-12-19 16:30:45 UTC - David Kjerrumgaard:
----
2018-12-19 16:31:19 UTC - David Kjerrumgaard:
----
2018-12-19 16:49:06 UTC - Ganga Lakshmanasamy: @Ali Ahmed Is there any unique constraint checks similar to sql unique available for pulsar topics? We basically want to avoid duplicate data being put up into topics from 2 different threads. Is that configurable?
----
2018-12-19 17:28:32 UTC - Mike Card: Hey everyone- I am getting some unusual messages in my log when I try to create partitioned topics in my code using the Pulsar admin client. My code that creates the topics looks like this:
```
try {admin.topics().createPartitionedTopic(FULL_MASTER_QUEUE_TOPIC_NAME, NUM_PARTITIONS); } catch (Throwable t) { _log.error("DefaultDatabus.constructor: caught exception " + t.getClass().getName() + " creating " + FULL_MASTER_QUEUE_TOPIC_NAME + ", message = " + t.getMessage());}
try {admin.topics().createPartitionedTopic(FULL_RESOLVER_QUEUE_TOPIC_NAME, NUM_PARTITIONS); } catch (Throwable t) { _log.error("DefaultDatabus.constructor: caught exception " + t.getClass().getName() + " creating " + FULL_RESOLVER_QUEUE_TOPIC_NAME + ", message = " + t.getMessage());}
try {admin.topics().createPartitionedTopic(FULL_RESOLVER_RETRY_QUEUE_TOPIC_NAME, NUM_PARTITIONS); } catch (Throwable t) { _log.error("DefaultDatabus.constructor: caught exception " + t.getClass().getName() + " creating " + FULL_RESOLVER_RETRY_QUEUE_TOPIC_NAME + ", message = " + t.getMessage());}
```
----
2018-12-19 17:29:17 UTC - Mike Card: The error messages I am getting look like this:
```
WARN [2018-12-19 16:21:41,643] org.apache.pulsar.client.admin.internal.BaseResource: [<http://emodb-pulsar-databus-pulsar-cluster-1:8080/admin/v2/persistent/public/default/master-queue/partitions>] Failed to perform http put request: java.net.SocketException: Unexpected end of file from server
ERROR [2018-12-19 16:21:41,643] com.bazaarvoice.emodb.databus.core.DefaultDatabus: DefaultDatabus.constructor: caught exception org.apache.pulsar.client.admin.PulsarAdminException creating public/default/master-queue, message = java.net.SocketException: Unexpected end of file from server
WARN [2018-12-19 16:21:41,649] org.apache.pulsar.client.admin.internal.BaseResource: [<http://emodb-pulsar-databus-pulsar-cluster-1:8080/admin/v2/persistent/public/default/resolver-queue/partitions>] Failed to perform http put request: java.net.SocketException: Unexpected end of file from server
ERROR [2018-12-19 16:21:41,649] com.bazaarvoice.emodb.databus.core.DefaultDatabus: DefaultDatabus.constructor: caught exception org.apache.pulsar.client.admin.PulsarAdminException creating public/default/resolver-queue, message = java.net.SocketException: Unexpected end of file from server
WARN [2018-12-19 16:21:41,654] org.apache.pulsar.client.admin.internal.BaseResource: [<http://emodb-pulsar-databus-pulsar-cluster-1:8080/admin/v2/persistent/public/default/resolver-retry-queue/partitions>] Failed to perform http put request: java.net.SocketException: Unexpected end of file from server
ERROR [2018-12-19 16:21:41,654] com.bazaarvoice.emodb.databus.core.DefaultDatabus: DefaultDatabus.constructor: caught exception org.apache.pulsar.client.admin.PulsarAdminException creating public/default/resolver-retry-queue, message = java.net.SocketException: Unexpected end of file from server
```
----
2018-12-19 17:29:34 UTC - Mike Card: Is there something I need to check in my config?
----
2018-12-19 17:44:14 UTC - Mike Card: @David Kjerrumgaard or @Matteo Merli have either of you guys ever seen this error using the admin client? What should I be looking for to resolve it?
----
2018-12-19 17:50:45 UTC - Sanjeev Kulkarni: @Ganga Lakshmanasamy Pulsar does not support sql unique type feature. Because, at its heart, Pulsar is a pub-sub/log layer and not really a database. Thus apart from scheme enforcement, very few additional checks are done during the time of publishing. This results in writes being blazing fast, but sql unique type checks aren’t done
----
2018-12-19 17:55:48 UTC - Sanjeev Kulkarni: @Mike Card This looks like something wrong with the connectivity. Are you sure Pulsar is reachable from the machine where your client runs?
----
2018-12-19 17:56:40 UTC - Mike Card: Yes, all of the other pulsar activity conducted via the Pulsar client API (like sending and receiving messages) works fine
----
2018-12-19 17:57:04 UTC - Mike Card: The only issues are these calls from the admin client
----
2018-12-19 17:57:27 UTC - Mike Card: I am wondering if my problem is that the admin client doesn't work well going through a load balancer
----
2018-12-19 17:57:48 UTC - Mike Card: I have a listener for port 8080 but maybe that's not the way to do the admin client
----
2018-12-19 18:01:32 UTC - Mike Card: The Pulsar client calls seem to work OK being sent through an ELB, I have listeners for both 6650 and 8080
----
2018-12-19 18:29:53 UTC - Mike Card: @Sanjeev Kulkarni @David Kjerrumgaard @Matteo Merli Hey guys I figured it out, it is that the Admin Java client does not like getting a URL that talks to a ELB as opposed to directly to a broker node. Previously, the URL I had been using to build the admin client was like this:
<http://emodb-pulsar-databus-pulsar-cluster-1:8080>
where emodb-pulsar-databus-pulsar-cluster-1 is a Route53 alias that points to my ELB, which has a listener for port 8080. When I do it this way I get the unexpected end-of-file exception on my calls to create partitioned topics per my earlier message.
However, if I change the URL to instead be the IP address of one of the broker nodes like this:
<http://10.110.78.37:8080>
then it works fine and the topics get created as expected.
This strikes me as a bug, the regular pulsar client can talk via an ELB but the admin client can't?
----
2018-12-19 19:08:10 UTC - Sanjeev Kulkarni: @Mike Card indeed. Could you please file an issue on github?
----
2018-12-19 19:26:09 UTC - Mike Card: @Sanjeev Kulkarni OK opened #3220 <https://github.com/apache/pulsar/issues/3220>
+1 : David Kjerrumgaard, Ali Ahmed
----
2018-12-19 21:39:28 UTC - Irfan: Hi all. I'm trying to run a simple tiered storage example on my local where I connect the standalone broker to a S3 bucket. However when I start up the standalone broker I get the following exception: `java.io.IOException: No offloader found for driver 'S3'. Please make sure you dropped the offloader nar packages under ${PULSAR_HOME}/offloaders.` What do I need to do fix this?
----
2018-12-20 00:11:18 UTC - Sijie Guo: That would be awesome!
----
2018-12-20 06:00:48 UTC - Ganga Lakshmanasamy: ok Thanks
----
2018-12-20 06:01:38 UTC - Ganga Lakshmanasamy: @Sanjeev Kulkarni Do you have any inputs for this query?
----
2018-12-20 06:57:19 UTC - Roger Yates: Hi all.
*What?* I'm trying to publish from a local pulsar client to a partitioned topic in the cloud, but my client machine sits behind a corporate proxy.
*What's the symptom?* I get a failure to connect with `org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to `... - the request isn't going via the corporate proxy, but it needs to.
*What have you tried?* I've tried various methods of configuring a proxy (JVM ARGS, system properties, for `socksProxyHost/Port` & `http.ProxyHost/Port` & tried creating a `ProxySelector` and instantiating on the client, which initializes but never gets its select() method called by the `Producer/PulsarClient`) , but none seem to have any effect.
I know the connectivity is there (firewalls have been burnt), as I have installed Proxifier to intercept and route to the proxy, and the messages were sent successfully; however, I can't install Proxifier on our servers so I need another way.
I haven't seen anything in the documentation to be able to specify a proxy on a ClientBuilder or Producer but I imagine this is a relatively common scenario in a corporate environment.
*Pulsar version:* 2.2.0
*So what's the question?*
Has anyone else been able to configure Pulsar to publish to an externally hosted topic from behind a corporate proxy?
If so, what did you need to do? Can you point me the right direction?
Thanks in advance.
----