You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/02/14 09:11:02 UTC
Slack digest for #general - 2020-02-14
2020-02-13 09:17:54 UTC - Abhilash Mandaliya: hello all
I am getting an exception for topic having the schema:
`Caused by: org.apache.commons.lang3.SerializationException: Failed at fetching schema info for EMPTY`
`at org.apache.pulsar.client.impl.schema.StructSchema.getSchemaInfoByVersion(StructSchema.java:182) ~[org.apache.pulsar-pulsar-client-original-2.5.0.jar:2.5.0]`
`at org.apache.pulsar.client.impl.schema.generic.GenericJsonSchema.loadReader(GenericJsonSchema.java:52) ~[org.apache.pulsar-pulsar-client-original-2.5.0.jar:2.5.0]`
`at org.apache.pulsar.client.impl.schema.StructSchema$1.load(StructSchema.java:72) ~[org.apache.pulsar-pulsar-client-original-2.5.0.jar:2.5.0]`
`at org.apache.pulsar.client.impl.schema.StructSchema$1.load(StructSchema.java:69) ~[org.apache.pulsar-pulsar-client-original-2.5.0.jar:2.5.0]`
`at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[java-instance.jar:?]`
`at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2276) ~[java-instance.jar:?]`
`at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) ~[java-instance.jar:?]`
`at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) ~[java-instance.jar:?]`
`... 11 more`
But the schema with version 0 does exist. I am getting this response while checking via pulsar-admin:
`./pulsar-admin schemas get test-topic`
response:
`{`
`"version": 0,`
`"schemaInfo": {`
`"name": "test-topic",`
`"schema": {`
`"name": "MyClass",`
`"type": "record",`
`"namespace": "com.acme.avro",`
`"fields": [`
`{`
`"name": "name",`
`"type": "string"`
`}`
`]`
`},`
`"type": "JSON",`
`"properties": {}`
`}`
`}`
Can anyone help here?
----
2020-02-13 09:43:41 UTC - Sijie Guo: If the pulsar manager is spinning, it usually means the pulsar manager can’t connect to your pulsar cluster. you need to ensure the network connectivity between pulsar manager and your pulsar cluster.
----
2020-02-13 09:46:34 UTC - Sijie Guo: a github issue with your steps led to this error will be much better.
the error seems indicating it is attempting to read a schema of empty version. but I have no idea how do you produce and consume messages.
----
2020-02-13 10:17:16 UTC - Eugen: For me it's only for the cluster view, not for other views, which work fine, so there would seem to be a connection between manager and standalone "cluster"...
----
2020-02-13 11:22:26 UTC - Miroslav Prymek: Hello, I'm experiencing a strange problem when trying to run bookies separated from brokers (on different machine) and also use them as a functions state storage.
options added to `bookkeeper.conf` :
```zkServers=zk-1:2181,zk-2:2181,zk-3:2181
extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent```
bookie started with:
```bin/pulsar bookie```
in the official docker image (`apachepulsar/pulsar:2.5.0`)
bookie log:
```bookie_1 | 11:17:01.151 [DLM-/stream/storage-OrderedScheduler-1-0-SendThread(localhost:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
bookie_1 | 11:17:01.151 [DLM-/stream/storage-OrderedScheduler-1-0-SendThread(localhost:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: localhost/127.0.0.1:2181: Connection refused```
It seems to me that `zkServers` option is not propagated into stream storage service and the default (`localhost:2181`) is used. Normal Pulsar topics storage (bookie at port 3181) works well.
Is there some option I missed? Should I force the `zkServers` option with some environment variable? Any other possible workaround?
Thanks for any suggestion.
----
2020-02-13 11:46:45 UTC - Abhilash Mandaliya: @Sijie Guo <https://github.com/apache/pulsar/issues/6321>
----
2020-02-13 13:42:49 UTC - Tilden: Hi All , we want to understand the requirement/ suggestion from Apache Pulsar on the designing the storage . we are planning to use CePH Storage from OpenStack, we want to know, what is the recommended disk I/O parameter?
----
2020-02-13 13:58:29 UTC - Vladimir Shchur: Can't run 2.5.0 bookie in k8s with default helm config, help is appreciated. `Error: Could not find or load main class`
----
2020-02-13 14:02:46 UTC - Konstantinos Papalias: not sure if this helps: <https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1580739667152700>
----
2020-02-13 14:06:22 UTC - Roman Popenov: Also make sure that when you save the config files, it’s UTF-8 encoding
----
2020-02-13 14:07:51 UTC - Roman Popenov: Otherwise, please try
``` BOOKIE_MEM: "\"-Xms128m -Xmx256m -XX:MaxDirectMemorySize=256m -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -verbosegc -XX:G1LogLevel=finest\""
BOOKIE_GC: "\"-XX:+UseG1GC -XX:MaxGCPauseMillis=10\""```
settings for bookies
----
2020-02-13 14:09:05 UTC - Roman Popenov: And I think you will need to add `bin/apply-config-from-env.py conf/bkenv.sh` command when the container is starting
----
2020-02-13 14:47:06 UTC - Pavel Tishkevich: @Pavel Tishkevich has joined the channel
----
2020-02-13 14:59:31 UTC - Pavel Tishkevich: Hi All!
Are there any recommendations about number of brokers/ZK/BK instances in cluster depending on load in terms of number of topics, in/out rate, etc?
For example: how many brokers do I need to deploy in cluster to tolerate single broker failure w/o experiencing serious latency blip? (Considering that overall number of topics in cluster is about 150k, publish/delivery rates is about 30k on average)
----
2020-02-13 17:49:10 UTC - Vladimir Shchur: @Roman Popenov thank you! Adding BOOKIE_MEM and BOOKIE_GC helped!
bananadance : Roman Popenov
----
2020-02-13 17:53:12 UTC - Sijie Guo: replied.
----
2020-02-13 18:40:42 UTC - Kenny: @Kenny has joined the channel
----
2020-02-13 22:03:50 UTC - Vimal Kumar: @Vimal Kumar has joined the channel
----
2020-02-13 23:49:31 UTC - Dzmitry Kazimirchyk: @Dzmitry Kazimirchyk has joined the channel
----
2020-02-14 02:38:39 UTC - Eugen: I'm trying to make sense of seemingly conflicting statements in the [load-balance docs](<https://pulsar.apache.org/docs/en/administration-load-balance/>):
> You can "unload" a topic in Pulsar with admin operation. *Unloading means* to close the topics, release ownership and *reassign the topics* to a new broker, based on current load.
> [..]
> *Unloading a topic has no effect on the assignment*, but just closes and reopens the particular topic:
So which one is right?
----
2020-02-14 02:47:58 UTC - Eugen: And one more question: How can unloaded topics be reassigned to another bundle (without splitting bundles), if this holds:
> Topics are assigned to a particular bundle by taking the hash of the topic name and checking in which bundle the hash falls into.
As the topic name does not change, the hash will not change, hence the bundle should not change either. What am I missing?
----
2020-02-14 03:42:26 UTC - Sijie Guo: “unload” doesn’t change the assignment - it basically close the topic. it can be “re-assigned” after unloaded if the load manager redistributes the bundles for load balancing.
+1 : Eugen
----
2020-02-14 04:26:18 UTC - Aaron Zhuo: @Aaron Zhuo has joined the channel
----
2020-02-14 04:28:35 UTC - Abhilash Mandaliya: thanks. I will try to produce a message via client code
----
2020-02-14 05:48:13 UTC - Kelvin Sajere: I experience the same thing @Eugen. The cluster page keeps spining. But every thing else seem to work
----
2020-02-14 06:41:17 UTC - Lari Hotari: Has anyone noticed issues with org.apache.pulsar.client.api.Reader#hasMessageAvailableAsync in Pulsar 2.5.0 ? The behavior has changed in a specific usecase in compared to 2.4.2 . I'm using a specified startMessageId with startMessageIdInclusive . For my usecase hasMessageAvailableAsync always returns false in 2.5.0 . It works in 2.4.2 as expected. I have a custom built Project Reactor adapter over the async API and since it's not possible to cancel a CompletableFuture, I use a pattern where before reading a message with readNextAsync, it is first checked with hasMessageAvailableAsync that there's a message available. I don't have an isolated test case for reproducing the failure yet.
----