You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/08/07 09:11:05 UTC
Slack digest for #general - 2020-08-07
2020-08-06 12:07:42 UTC - Alan Hoffmeister: is pulsar recommended as a permanent event storage? I'm trying to setup some event sourcing structure but I would like to avoid storing my event log inside mysql
----
2020-08-06 14:31:18 UTC - Pramiti Guha: @Pramiti Guha has joined the channel
----
2020-08-06 15:07:03 UTC - David Kjerrumgaard: @Alan Hoffmeister I would definitely say that Pulsar is a good candidate for permanent event storage, particularly with its tiered-storage capabilities.
+1 : Alan Hoffmeister, Rich Adams, Chris Hansen
----
2020-08-06 15:09:45 UTC - David Kjerrumgaard: If you have a scenario where you have older noBK nodes re-joining the cluster, then you will end up with some over-replicated segments (which isn't bad) and more storage capacity. I can't envision a scenario where this would result in unwanted behavior. Are you experiencing any?
----
2020-08-06 15:20:22 UTC - Feanil: Hey all, I'm trying to wrap my head around pulsar schema evolution with JSON schemas(in python). I can't seem to find good docs on what the format of the schema is, that is actually sent to the pulsar registry. Is it an implementation of the `json-schema` spec? Is any JSON schema is valid? or is it something more restrictive? In particular it seems like the python bindings don't seem to generate the schema with a default value, is this on purpose?
----
2020-08-06 15:32:30 UTC - Bre Gielissen: Quick question: can you clarify the error "Cannot delete non empty namespace"? I was assuming that it meant that there were topics underneath but that either means the documentation is wrong in your REST API documentation or I actually need to do something else to truly "empty" the namespace. Thanks!
----
2020-08-06 16:03:01 UTC - David Kjerrumgaard: @Feanil I spent a lot of time focusing on Schemas in my upcoming book about Pulsar because i found the public documentation to be lacking. I gave an example using Avro, and found it to be quite easy to use. There is also avro support for Python
----
2020-08-06 16:22:54 UTC - Addison Higham: are you using async or sync APIs? for performance, the biggest thing you can do first is make sure you are using async producer and consumer and to enable batching
+1 : charles
----
2020-08-06 16:23:03 UTC - Addison Higham: @charles ^^
----
2020-08-06 16:24:48 UTC - Alan Broddle: Has anyone tried to use the Prometheus metrics in DataDog? Our corporate preferred monitoring environment is DataDog. We have enabled the DataDog Zookeeper and DataDog OpenMetrics integrations from our Pulsar cluster, and the data is seen by DataDog. The issue is DataDog does not ‘like’ the complexity of the metrics Prometheus/Pulsar creates. DD does not understand metrics that are formatted with comma separated values in the metric conditionals.
pulsar_rate_in{cluster=“Test”,namespace=“public/default”,topic=“<persistent://public/default/my-topic2>”} 99.999 1596728053329
More details if needed…
Basically anted to know if anyone has tried this yet?
----
2020-08-06 16:29:28 UTC - Addison Higham: @Alan Broddle I have used datadog in the past, datadog with the openmetrics integration should do fine with the labels. What we had to do though was add annotations for metrics that don't have type metadata. In most promotheus scrapers, if a metric doesn't have type info, it just gets interpreted as a gauge, but datadog is a bit more picky and only will consumer metrics that it has type info for, either by the type annotations provided by the endpoint OR if you add your own annotations, as an example:
```<http://ad.datadoghq.com/broker.instances|ad.datadoghq.com/broker.instances>: '[{"prometheus_url":"<http://%%host%>%:8080/metrics","namespace":"pulsar","metrics":["*"],"type_overrides":{"pulsar_topics_count":"gauge","pulsar_subscriptions_count":"gauge"}}'```
----
2020-08-06 16:30:04 UTC - Addison Higham: :thinking_face: I don't think that doc is correct
----
2020-08-06 16:51:09 UTC - Vil: Ok, thanks for your reply @Addison Higham. Not sure I understand everything in your reply (what to do with acks etc.), so I use my own words to summarize what you said. Let me know if that’s correct.
1. I can attempt to make effectively once work with pulsar functions but it requires extra work on my end (and here I do not know exactly what I would have to do).
2. Better option is to not use pulsar functions for effectively once but instead use flink.
Is this a good summary?
If it is my question is are there any plans to make pulsar functions ‘just work’ for effectively once? I mean, roadmap?
----
2020-08-06 16:53:44 UTC - Addison Higham: for sources/sinks particularity, pulsar functions by themselves, when you use just a simple input and output topic already "just work"
----
2020-08-06 16:55:29 UTC - Vil: to use an example: input topic > pulsar function > output topic, would just work? and if yes (nice!) I do not yet understand in which case it would not work :slightly_smiling_face:
----
2020-08-06 16:56:06 UTC - Vil: or do you mean input topic > pulsar functions work, and pulsar function > output topic works, but not the full line of input > pulsar function > output ?
----
2020-08-06 16:56:47 UTC - Addison Higham: the first one
----
2020-08-06 16:58:04 UTC - Addison Higham: So yes, with a simple input > function > output, you get effectively once. But, for that, they can't handle as much throughput, as we can't use batch any other techniques and must ack per message
----
2020-08-06 17:29:11 UTC - Brandon Whitehead: can someone help me actually send a message on the localhost:6650?
----
2020-08-06 17:29:33 UTC - Brandon Whitehead: I'm trying to make a go program that sends a simple message on the actual localhost
----
2020-08-06 17:29:56 UTC - Brandon Whitehead: but despite it seeming to work in the backend, I'm not sure how to send a message on the frontend
----
2020-08-06 17:30:32 UTC - Feanil: @Brandon Whitehead Are you following the client example(<https://pulsar.apache.org/docs/en/next/client-libraries-go/#producers>) or something else?
----
2020-08-06 17:31:43 UTC - Brandon Whitehead: I am
----
2020-08-06 17:32:32 UTC - Feanil: Do you see any output, the python client is pretty verbose, I imagine the Go one is similar
----
2020-08-06 17:32:44 UTC - Brandon Whitehead: this is the code I have
----
2020-08-06 17:32:53 UTC - Brandon Whitehead: ```package main;
import (
"log"
"fmt"
"time"
"context"
"<http://github.com/apache/pulsar-client-go/pulsar|github.com/apache/pulsar-client-go/pulsar>"
)
func main() {
client, err := pulsar.NewClient(pulsar.ClientOptions{
URL: "<pulsar://localhost:6650>",
OperationTimeout: 60 * time.Second,
ConnectionTimeout: 60 * time.Second,
})
if err != nil {
log.Fatalf("Could not instantiate Pulsar client: %v", err)
} else {
log.Printf("Success!")
}
defer client.Close()
producer, err1 := client.CreateProducer(pulsar.ProducerOptions{
Topic: "my-topic",
Name: "test-producer",
})
if err1 != nil {
log.Fatalf("Could not connect to client: %v", err1)
} else {
log.Printf("Success!")
}
_, err2 := producer.Send(context.Background(), &pulsar.ProducerMessage{
Payload: []byte("hello"),
})
defer producer.Close()
if err2 != nil {
fmt.Println("Failed to publish message", err2)
} else {
log.Printf("Success!")
}
fmt.Println("Published message")
consumer, err3 := client.Subscribe(pulsar.ConsumerOptions{
Topic: "topic-1",
SubscriptionName: "my-sub",
Type: pulsar.Shared,
})
if err3 != nil {
log.Fatal(err3)
}
defer consumer.Close()
for i := 0; i < 10; i++ {
msg, err4 := consumer.Receive(context.Background())
if err4 != nil {
log.Fatal(err4)
} else {
log.Printf("Success!")
}
log.Printf("Received message msgId: %#v -- content: '%s'\n",
msg.ID(), string(msg.Payload()))
consumer.Ack(msg)
}
if err5 := consumer.Unsubscribe(); err5 != nil {
log.Fatal(err5)
}
}```
----
2020-08-06 17:32:59 UTC - Feanil: Also, have you confirmed that the pulsar server is accepting connections?
----
2020-08-06 17:33:18 UTC - Brandon Whitehead: from what I can see, the go program connected
----
2020-08-06 17:33:22 UTC - Feanil: `telnet localhost 6650` and seeing if that successfully establishes a connection
----
2020-08-06 17:33:31 UTC - Brandon Whitehead: where do I enter that?
----
2020-08-06 17:33:36 UTC - Brandon Whitehead: I'm using docker
----
2020-08-06 17:33:37 UTC - Vil: ok, thank you again @Addison Higham
----
2020-08-06 17:34:45 UTC - Feanil: did you bind the port to your localhost? How are you using docker(with docker-compose, or manually bringing up multiple containers?)
----
2020-08-06 17:35:04 UTC - Brandon Whitehead: I'm just using the tutorial on the website
----
2020-08-06 17:35:26 UTC - Brandon Whitehead: so if I got the terminology right, I think it's only one container?
----
2020-08-06 17:36:22 UTC - Bre Gielissen: I was wondering. That is unfortunate. That would make it so much easier.
----
2020-08-06 17:36:28 UTC - Brandon Whitehead: <https://pulsar.apache.org/docs/en/next/standalone-docker/>
----
2020-08-06 17:43:03 UTC - Brandon Whitehead: so in my go program, I make a client, then a producer
----
2020-08-06 17:43:25 UTC - Brandon Whitehead: how do I get the localhost to be a consumer so they get the producer's message?
----
2020-08-06 17:44:16 UTC - Feanil: gotcha so you should be able to run that telnet command on your local machine in a shell
----
2020-08-06 17:44:24 UTC - Feanil: you shouldn't need to run it in the container
----
2020-08-06 17:44:47 UTC - Brandon Whitehead: the telnet command doesn't work
----
2020-08-06 17:44:51 UTC - Brandon Whitehead: in powershell
----
2020-08-06 17:45:03 UTC - Brandon Whitehead: doesn't recognize the telnet command at all
----
2020-08-06 17:45:19 UTC - Feanil: @David Kjerrumgaard do you know what the format of the schema you upload is supposed to be? I'm trying to upload the schema but it's not exactly clear what the format of the file is supposed to be
----
2020-08-06 17:45:49 UTC - Ebere Abanonu: @Brandon Whitehead if the pulsar is running and you cannot connect then try 127.0.0.1
----
2020-08-06 17:46:20 UTC - Feanil: ah didn't realize you were on windows
----
2020-08-06 17:46:24 UTC - Brandon Whitehead: I think pulsar is running though
----
2020-08-06 17:46:29 UTC - Brandon Whitehead: docker says it connected
----
2020-08-06 17:46:39 UTC - Feanil: try:
```Test-NetConnection -ComputerName localhost -Port 6650```
----
2020-08-06 17:46:44 UTC - Brandon Whitehead: and the /metrics page for the localhost:8080 says it did too
----
2020-08-06 17:47:45 UTC - Brandon Whitehead: where do I figure out my computername?
----
2020-08-06 17:48:07 UTC - Feanil: it's just localhost `-ComputerName` is the name of the parameter
----
2020-08-06 17:48:21 UTC - Brandon Whitehead: whoops
----
2020-08-06 17:48:33 UTC - Feanil: you can also try `127.0.0.1` if localhost doesn't connect
----
2020-08-06 17:48:46 UTC - Brandon Whitehead: okay I got this
----
2020-08-06 17:48:58 UTC - Brandon Whitehead: ```ComputerName : localhost RemoteAddress : ::1 RemotePort : 6650 InterfaceAlias : Loopback Pseudo-Interface 1
SourceAddress : ::1
TcpTestSucceeded : True```
----
2020-08-06 17:49:20 UTC - Brandon Whitehead: what exactly does this mean?
----
2020-08-06 17:49:33 UTC - Feanil: ok, so it looks like connection is fine, in that case maybe start with the simple example code from the client docs
----
2020-08-06 17:49:59 UTC - Brandon Whitehead: and what exactly do I use to send messages to that localhost so I can actually see them on my computer when I reach the page?
----
2020-08-06 17:50:28 UTC - Brandon Whitehead: I'm new with this whole pulsar thing and I'm looking through the architecture and it's confusing me about what actually happens
----
2020-08-06 17:50:54 UTC - Brandon Whitehead: so I make a client in my go program that is tied to the localhost, then what?
----
2020-08-06 17:51:16 UTC - Brandon Whitehead: do I make a producer to make a message, and then make a consumer to get that message? or do I use a reader?
----
2020-08-06 17:52:19 UTC - Brandon Whitehead: cause the go client tutorial is not clear about what order this code needs to be in, or if it needs to be multiple different files running concurrently
----
2020-08-06 17:52:20 UTC - Feanil: you need a producer in order to make the messages and then you can consume the messages with a consumer
----
2020-08-06 17:52:40 UTC - Brandon Whitehead: and when I consume the messages, shouldn't they show up on the localhost?
----
2020-08-06 17:53:04 UTC - Feanil: what do you mean by "show up" on localhost?
----
2020-08-06 17:53:37 UTC - Brandon Whitehead: like when I actually go to the actual page on my browser, currently nothing shows up
----
2020-08-06 17:53:39 UTC - Feanil: producers put messages into pulsar, and then consumers take messages out, once the consumer takes the message out, they can print it to screen or proccess it in some other way
----
2020-08-06 17:53:50 UTC - Feanil: which page are you going to?
----
2020-08-06 17:54:12 UTC - Brandon Whitehead: `<http://localhost:6650/>`
----
2020-08-06 17:54:32 UTC - Feanil: right, that's not an HTTP endpoint, that's the endpoint your producer/consumer talk to
----
2020-08-06 17:54:44 UTC - Brandon Whitehead: okay?
----
2020-08-06 17:54:50 UTC - Brandon Whitehead: what does that mean exactly?
----
2020-08-06 17:54:56 UTC - Feanil: if you want to get a message back out of pulsar, you need a consumer process
----
2020-08-06 17:55:16 UTC - Brandon Whitehead: well I made a consumer in my go program like the tutorial said
----
2020-08-06 17:55:42 UTC - Brandon Whitehead: and it's supposed to receive the message from the topic the producer made, and then display it
----
2020-08-06 17:55:50 UTC - Brandon Whitehead: at least if I'm reading the code right
----
2020-08-06 17:56:26 UTC - Brandon Whitehead: here is the code
----
2020-08-06 17:56:30 UTC - Brandon Whitehead: ```package main;
import (
"log"
"fmt"
"time"
"context"
"<http://github.com/apache/pulsar-client-go/pulsar|github.com/apache/pulsar-client-go/pulsar>"
)
func main() {
client, err := pulsar.NewClient(pulsar.ClientOptions{
URL: "<pulsar://localhost:6650>",
OperationTimeout: 60 * time.Second,
ConnectionTimeout: 60 * time.Second,
})
if err != nil {
log.Fatalf("Could not instantiate Pulsar client: %v", err)
} else {
log.Printf("Success!")
}
defer client.Close()
producer, err1 := client.CreateProducer(pulsar.ProducerOptions{
Topic: "my-topic",
Name: "test-producer",
})
if err1 != nil {
log.Fatalf("Could not connect to client: %v", err1)
} else {
log.Printf("Success!")
}
_, err2 := producer.Send(context.Background(), &pulsar.ProducerMessage{
Payload: []byte("hello"),
})
defer producer.Close()
if err2 != nil {
fmt.Println("Failed to publish message", err2)
} else {
log.Printf("Success!")
}
fmt.Println("Published message")
consumer, err3 := client.Subscribe(pulsar.ConsumerOptions{
Topic: "my-topic",
SubscriptionName: "my-sub",
Type: pulsar.Shared,
})
if err3 != nil {
log.Fatal(err3)
}
defer consumer.Close()
for i := 0; i < 10; i++ {
msg, err4 := consumer.Receive(context.Background())
if err4 != nil {
log.Fatal(err4)
} else {
log.Printf("Success!")
}
fmt.Printf("Received message msgId: %#v -- content: '%s'\n",
msg.ID(), string(msg.Payload()))
consumer.Ack(msg)
}
if err5 := consumer.Unsubscribe(); err5 != nil {
log.Fatal(err5)
}
}```
----
2020-08-06 17:56:30 UTC - Feanil: it seems like you don't have a clear undrestanding of TCP vs HTTP, is that right?
----
2020-08-06 17:56:39 UTC - Brandon Whitehead: I guess not
----
2020-08-06 17:56:54 UTC - Brandon Whitehead: I'm sorry, but I'm really lost
----
2020-08-06 17:57:08 UTC - Feanil: <https://www.extrahop.com/company/blog/2018/tcp-vs-http-differences-explained/>
----
2020-08-06 17:57:16 UTC - Brandon Whitehead: all I want is to show a simply text line saying "hello" on the localhost
----
2020-08-06 17:57:45 UTC - Feanil: if you're trying to understand how the messaging system works you'll probably want a good handle on the basics of how the communication is happening
----
2020-08-06 17:58:09 UTC - Feanil: in general though, message buses are not super useful if you are in the same process
----
2020-08-06 17:58:19 UTC - Feanil: they become more useful when trying to communicate across processes
----
2020-08-06 17:59:11 UTC - Brandon Whitehead: so TCP is about sending out data, and HTTP is about reading that data?
----
2020-08-06 17:59:27 UTC - Brandon Whitehead: is that the basic gist?
----
2020-08-06 18:01:03 UTC - Brandon Whitehead: if that's the case, then the localhost should be trying to receive and read the data from my go program right?
----
2020-08-06 18:22:16 UTC - Evan Furman: Does anyone have prometheus/grafana configs they can share that are NOT for kubernetes? Looking for an example of how to setup configs running on ec2
----
2020-08-06 18:26:16 UTC - Prashanth Tirupachur Vasanthakrishnan: <http://pulsar.apache.org/docs/en/reference-metrics/#monitor> The “apache-pulsar-grafana-dashboard” link has grafana dashboards for k8 & on-prem.
----
2020-08-06 18:26:55 UTC - Brandon Whitehead: so I haven't really gotten any closer to getting a message on the localhost that I can see on the browser
----
2020-08-06 18:27:42 UTC - Brandon Whitehead: is there something, I should be doing within the go program?
----
2020-08-06 18:28:23 UTC - Brandon Whitehead: the topic I created with the producer within the go program works fine
----
2020-08-06 18:28:59 UTC - Brandon Whitehead: and the consumer says it's received the message via the powershell log, but I'm not sure how to translate that Payload from the Producer onto the screen
----
2020-08-06 18:42:55 UTC - David Kjerrumgaard: There is a tool for manually downloading and uploading schemas directly to Pulsar, along with some examples here. <https://pulsar.apache.org/docs/en/schema-manage/#upload-a-schema>
----
2020-08-06 18:49:58 UTC - Evan Furman: Thanks @Prashanth Tirupachur Vasanthakrishnan! I got it going but seeing some really odd stuff. I have 4 bookies…
----
2020-08-06 19:02:57 UTC - Addison Higham: I agree, I think there was some discussion on making an API for that...
----
2020-08-06 19:16:16 UTC - Rattanjot Singh: ```19:15:22.336 [BookieJournal-3181] ERROR org.apache.bookkeeper.bookie.BookieCriticalThread - Uncaught exception in thread BookieJournal-3181 and is exiting!
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 4294967296, max: 4294967296)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:726) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:681) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:758) ~[io.netty-netty-buffer-4.1.43.Final.jar:4.1.43.Final]
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:734) ~[io.netty-netty-buffer-4.1.43.Final.jar:4.1.43.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:245) ~[io.netty-netty-buffer-4.1.43.Final.jar:4.1.43.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:227) ~[io.netty-netty-buffer-4.1.43.Final.jar:4.1.43.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:147) ~[io.netty-netty-buffer-4.1.43.Final.jar:4.1.43.Final]
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:342) ~[io.netty-netty-buffer-4.1.43.Final.jar:4.1.43.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187) ~[io.netty-netty-buffer-4.1.43.Final.jar:4.1.43.Final]
at org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl.newDirectBuffer(ByteBufAllocatorImpl.java:164) ~[org.apache.bookkeeper-bookkeeper-common-allocator-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl.newDirectBuffer(ByteBufAllocatorImpl.java:158) ~[org.apache.bookkeeper-bookkeeper-common-allocator-4.10.0.jar:4.10.0]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187) ~[io.netty-netty-buffer-4.1.43.Final.jar:4.1.43.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178) ~[io.netty-netty-buffer-4.1.43.Final.jar:4.1.43.Final]
at org.apache.bookkeeper.bookie.BufferedChannel.<init>(BufferedChannel.java:86) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.BufferedChannel.<init>(BufferedChannel.java:77) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.BufferedChannel.<init>(BufferedChannel.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Journal.lambda$getBufferedChannelBuilder$0(Journal.java:1129) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.JournalChannel.<init>(JournalChannel.java:175) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.JournalChannel.<init>(JournalChannel.java:114) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Journal.run(Journal.java:947) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
4.386: Total time for which application threads were stopped: 0.0001696 seconds, Stopping threads took: 0.0000834 seconds
4.387: Total time for which application threads were stopped: 0.0001677 seconds, Stopping threads took: 0.0001105 seconds
4.387: Total time for which application threads were stopped: 0.0001555 seconds, Stopping threads took: 0.0000935 seconds
19:15:22.338 [component-shutdown-thread] INFO org.apache.bookkeeper.replication.AutoRecoveryMain - Shutting down auto recovery: 0
19:15:22.338 [component-shutdown-thread] INFO org.apache.bookkeeper.replication.AutoRecoveryMain - Shutting down AutoRecovery
Heap```
getting this error in bookeeper
----
2020-08-06 19:17:26 UTC - Caleb Epstein: Need more memory for the JVM to run
----
2020-08-06 19:18:05 UTC - Rattanjot Singh: I am running on kubernetes. where to set that?
----
2020-08-06 19:18:50 UTC - Caleb Epstein: Not sure - I run on bare metal and its in conf/bkenv.sh, see BOOKIE_MEM
----
2020-08-06 19:19:55 UTC - Caleb Epstein: If you can pass in environment variables, you can set that and it should be honored by the script
----
2020-08-06 19:20:11 UTC - Caleb Epstein: I changed from 4G to 8G when I ran into the same failure
----
2020-08-06 19:20:37 UTC - Caleb Epstein: `BOOKIE_MEM=${BOOKIE_MEM:-${PULSAR_MEM:-"-Xms8g -Xmx8g -XX:MaxDirectMemorySize=8g"}}`
----
2020-08-06 19:21:14 UTC - Rattanjot Singh: let me try this
----
2020-08-06 19:21:15 UTC - julien: @julien has joined the channel
----
2020-08-06 19:29:01 UTC - julien: Hello, I am setting up bookkeeper with pulsar for production use, I am looking for filesystem recommendation, do you know if there is a documentation about that?
----
2020-08-06 19:54:26 UTC - Addison Higham: as in which type of filesystem and settings? XFS is what is used in the helm chart and i know it is also used in some of the larger bookkeeper installs, it helps with latencies. I have not seen much about additional tuning (like block size, etc)
The most important thing for performance in bookkeeper is fast journal disk
----
2020-08-06 19:56:27 UTC - julien: ok, thank you Addison
----
2020-08-06 20:11:53 UTC - Rattanjot Singh: yes it worked thanks
+1 : Caleb Epstein
----
2020-08-06 20:38:50 UTC - Caleb Epstein: Wow, 64 bookies. Curious how much traffic you're handling at that size.
----
2020-08-06 20:39:26 UTC - Caleb Epstein: 96% of 64 doesn't sound like 51 tho :slightly_smiling_face:
----
2020-08-06 21:00:44 UTC - Brandon Whitehead: so I have to wonder
----
2020-08-06 21:01:08 UTC - Brandon Whitehead: does the localhost that is made via docker connecting with pulsar actually for frontend use?
----
2020-08-06 21:02:36 UTC - Brandon Whitehead: cause I still run the go program, and I get no actual message from my consumer object in the program, yet I do get confirmation that the topic is running, it connected to the topic, and that it got A message
----
2020-08-06 21:02:48 UTC - Brandon Whitehead: but I have no way to show said message anywhere
----
2020-08-06 21:02:54 UTC - Brandon Whitehead: where does it get sent to?
----
2020-08-06 21:13:31 UTC - Brandon Whitehead: I'd just like to know how to look at a payload's message at all
----
2020-08-06 21:13:52 UTC - Brandon Whitehead: maybe then I can figure out how to transfer that to some simplistic HTML format for the localhost or something
----
2020-08-06 21:38:12 UTC - Axel Sirota: @Sijie Guo do you have any clues? I got this example from your tutorial :slightly_smiling_face:
----
2020-08-06 21:40:27 UTC - Sijie Guo: I am not sure if that example still works or not. The demo is 3-years old :slightly_smiling_face:
----
2020-08-06 21:40:36 UTC - Sijie Guo: I also need to try it to verify
----
2020-08-06 21:40:52 UTC - Axel Sirota: haha indeed! I couldn’t make it work :joy: is the client adapter still a thing??
----
2020-08-06 21:41:01 UTC - Axel Sirota: thanks for following!
----
2020-08-06 21:43:33 UTC - Axel Sirota: For reference, my app trying to use the adapter is in : <https://github.com/axel-sirota/handling-streaming-apache-pulsar/blob/master/src/main/resources/log4j2.xml>
I am trying to make a video course on Pulsar and how it makes a revolution in the pub-sub space
----
2020-08-06 21:43:42 UTC - Axel Sirota: So a video showing how to migrate will be super useful
----
2020-08-06 22:15:48 UTC - Shilpy: @Shilpy has joined the channel
----
2020-08-07 00:20:31 UTC - Brandon Whitehead: how do you use pulsarctl to force a consumer to unsubscribe?
----
2020-08-07 00:21:41 UTC - julien: Hello, is there a specific mechanism to backup data in bookkeeper? Do I need backup also metadata in Zookeeper or it is it possible to reconstruct it?
----
2020-08-07 00:24:34 UTC - Brandon Whitehead: right now, I'm trying to test my one consumer on my one topic by unsubscribing them from the topic, but using pulsarctl, I can't unsubscribe it because the consumer is still using the subscription
----
2020-08-07 00:41:53 UTC - Shilpy: Hi, i am getting below error in my service after the service has run for quite few hours : org.apache.pulsar.client.api.PulsarClientException$TimeoutException: The producer standalone-0-71 can not send message to the topic <TOPICNAME> with given timeout
----
2020-08-07 00:42:18 UTC - Shilpy: Can anybody help in this.
----
2020-08-07 03:21:25 UTC - Yuvaraj Loganathan: This is used by delayed message delivery. Do you publish messages with very huge delay ?
----
2020-08-07 05:22:35 UTC - Takahiro Hozumi: I skimmed the source code of compaction and found that the whole keys are held on memory using regular HashMap during compaction.
This implementation requires more memory than the total size of the key.
<https://github.com/apache/pulsar/blob/df19f261def8e66de213bff310a8423ae133ef58/pulsar-broker/src/main/java/org/apache/pulsar/compaction/TwoPhaseCompactor.java#L80>
----
2020-08-07 08:19:27 UTC - charles: @Addison Higham: I used batched sync :face_with_hand_over_mouth: .... switched to async now ... and the big numbers start reporting (seeing 100k+ msg/s) thanks for this reminder :+1:
----
2020-08-07 08:48:09 UTC - Arek R.: @Arek R. has joined the channel
----
2020-08-07 08:49:32 UTC - Arek R.: Hello, is it possible to load the data into pulsar when no consumer is running?
----