You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/09/12 09:11:03 UTC

Slack digest for #general - 2019-09-12

2019-09-11 09:56:33 UTC - wlkid: Hi guys, just want to know what's the order of the messages sent to same topic from multiple producers concurrently, from the point view of consumer?
----
2019-09-11 09:59:05 UTC - wlkid: is the order determined by the timestamp specified by the producers, or by the sequence the broker received the messages?
----
2019-09-11 11:47:41 UTC - Kirill Merkushev: so basically we are serializing messageId as long
```
    public static MessageId fromOffset(long offset) {
        return new MessageIdImpl(offset &gt;&gt;&gt; 28, offset &amp; 0x0F_FF_FF_FFL, -1);
    }

    public static long toOffset(MessageId messageId) {
        MessageIdImpl msgId = (MessageIdImpl) messageId;
        // Combine ledger id and entry id to form offset
        // Use less than 32 bits to represent entry id since it will get
        // rolled over way before overflowing the max int range
        return (msgId.getLedgerId() &lt;&lt; 28) | msgId.getEntryId();
    }
```
and keep that separately from the Pulsar, and after some time seeking that gives us exception about wrong seek

We discussed that in the <https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1567085799031200>
----
2019-09-11 11:48:36 UTC - Kirill Merkushev: but we didn’t get what’re the possible workarounds
----
2019-09-11 11:52:51 UTC - Kirill Merkushev: @Sijie Guo here is the rationale about ledger id question as well
----
2019-09-11 12:57:33 UTC - el akroudi abdessamad: I have a question
----
2019-09-11 12:58:22 UTC - el akroudi abdessamad: Any work to integrate pulsar with cloudeea stack, it will push pulsar in the top level and gain popularity
----
2019-09-11 12:58:27 UTC - el akroudi abdessamad: ??
----
2019-09-11 13:15:05 UTC - Nick Marchessault: Can someone clarify what ReceivedFailed and AcksFailed mean in the getStats method (<https://pulsar.apache.org/api/client/org/apache/pulsar/client/api/Consumer.html#getStats-->)? Is there a one to one correlation between AcksFailed and unacknowledged messages?
----
2019-09-11 13:32:49 UTC - Tarek Shaar: Thanks Yuva, my understanding was that ZK is an in memory store and it periodically stores the data in the transaction log file. So are you saying that each message that the producer sends to the broker gets stored in Book Keeper ledger log file and in the ZK disk before the ack is sent back? Can you please share the flow with me? This is critical for our company to understand
----
2019-09-11 13:34:27 UTC - Yuvaraj Loganathan: Zk stores only metadata.  But metadata is crtical. Actual flow of data does go through boookeeper and broker.
----
2019-09-11 13:35:21 UTC - Yuvaraj Loganathan: <https://pulsar.apache.org/docs/en/concepts-architecture-overview/>
----
2019-09-11 13:38:39 UTC - Tarek Shaar: Thanks Yuva. I have been through this document and it talks about how Book Keeper drives the persistence not ZK. I guess am trying to understand what information is stored in ZK for pulsar and at which stage etc...
----
2019-09-11 14:35:21 UTC - Matteo Merli: The order is determined by how they are received by broker
----
2019-09-11 14:49:28 UTC - Sandeep Kotagiri: @Mahesh two primary settings I have used are as what I wrote above. In bookkeeper.conf file add extraServerComponents= org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent and in functions_worker.yml add stateStorageServiceUrl: "<bk://localhost:4181>"
----
2019-09-11 14:51:08 UTC - Sandeep Kotagiri: For errors such as yours, you might find some different error in the bookkeeper log files. That might give a clue as of what is happening.
----
2019-09-11 15:40:12 UTC - Matteo Merli: Not directly. The unacked messages, from broker’s perspective, are all the messages that were pushed to one consumer and not acked yet.
----
2019-09-11 18:11:24 UTC - Subramaniyam: Hi All, I am trying to install pulsar-client in my npm (nodejs)

But i am getting the below error..
node-pre-gyp WARN Using request for node-pre-gyp https download
node-pre-gyp WARN Tried to download(404): <https://pulsar.apache.org/docs/en/client-libraries-cpp/libpulsar-v1.1.0-rc.0-node-v64-win32-x64.tar.gz>
node-pre-gyp WARN Pre-built binaries not found for pulsar-client@1.1.0-rc.0 and node@10.12.0 (node-v64 ABI, unknown) (falling back to source compile with node-gyp)
gyp WARN install got an error, rolling back install
gyp ERR! configure error
gyp ERR! stack Error: tunneling socket could not be established, statusCode=403

Looks like the above library is not there in the particular URL
Its roll backing total installation.
----
2019-09-11 19:50:44 UTC - Areeb Ahmed: @Areeb Ahmed has joined the channel
----
2019-09-11 20:25:16 UTC - Tarek Shaar: I am running internal stats on my pulsar topic. Here is snippet of what I get
----
2019-09-11 20:25:17 UTC - Tarek Shaar: "lastConfirmedEntry" : "11:7",
  "state" : "LedgerOpened",
  "ledgers" : [ {
    "ledgerId" : 7,
    "entries" : 5,
    "size" : 290,
    "offloaded" : false
  }, {
    "ledgerId" : 11,
    "entries" : 0,
    "size" : 0,
    "offloaded" : false
  } ],
----
2019-09-11 20:26:34 UTC - Tarek Shaar: Question is Ledger 11 shows no entries at all but last confirmed entry is 11:7. In other words I expect ledger 11 to have some entries but there are none. Am i missing something?
----
2019-09-12 03:54:52 UTC - Ali Ahmed: @Ali Ahmed set the channel topic: Pulsar graduated as a Top-Level project :tada: - Pulsar release 2.4.1 - <http://pulsar.apache.org/release-notes/#2.4.1>
----
2019-09-12 05:13:19 UTC - Vinuja: @Vinuja has joined the channel
----
2019-09-12 05:47:21 UTC - divyasree: Can you guide me to proceed further?
----
2019-09-12 06:09:44 UTC - Shishir Pandey: @Shishir Pandey has joined the channel
----
2019-09-12 06:45:13 UTC - Vinuja: I have deployed Pulsar on Azure Kubernetes service. I am testing Failover scenarios for Pulsar. As a scenario, I deleted the zookeeper and its pvc, Now I created a new zookeeper and a pvc  and initialized cluster metadata. Now, All the bookie, broker are all up. But the Cluster is not initialized properly. I am not able to see the cluster itself. Can you please help me in resolving this issue.
----
2019-09-12 06:47:40 UTC - Jerry Peng: @Tarek Shaar that is expected. The stats will show up when the ledger is closed.  So always expect the latest ledger to have empty
----
2019-09-12 06:48:51 UTC - wlkid: does it imply that there's no guarantee of FIFO? or, for a simpler case, consuming mysql binlog events needs strictly FIFO, will pulsar suitable for it?
----
2019-09-12 07:04:22 UTC - Walter: i have deployed pulsar in kubernetes with 1 broker pod(2cpu , 8Gb), 3 bookkeeper pod (each 2cpu, 8Gb), 3 zookeeper(1cpu, 2Gb)
i have given  more load with pulsar-perf tool but it's reached memory limit of broker means (7.9Gb). Then I waited for 2days to cool down broker pods memory. so why broker pod memory is not reducing?
----
2019-09-12 07:11:42 UTC - Zhenhao Li: looking at <https://www.slideshare.net/Odin_cool/fsharp-goodness-for-everyday-work>
----
2019-09-12 07:16:25 UTC - Zhenhao Li: I wonder what made the F# client is 1.5x as fast as Java
----
2019-09-12 08:09:40 UTC - Dennis Sehalic: <https://jack-vanlightly.com/blog/2019/9/4/a-look-at-multi-topic-subscriptions-with-apache-pulsar>
----
2019-09-12 08:50:16 UTC - Maksim Bazhenov: @Maksim Bazhenov has joined the channel
----
2019-09-12 08:53:32 UTC - Shishir Pandey: Hi all,
I am trying to solve the following problem - I want to place the messages into the queue sorted by the message key the following manner { message_key: {string: name, integer: id, message_created time}, message_value: { &lt;xml&gt; } } this is kind of to ensure that messages for the same name and id are next to each other, I want to do this because I need to process messages for a particular name and id that arrive within a specific interval (this is reasonably large - up to 7 days) together and I do not want to traverse the messages one at a time. I've been reading the documentation for Pulsar and it is not entirely clear to me if this can be achieved? If I understand it correctly the custom partitioning scheme would only put the message into a specific partition, and the documentation further says "The ordering of messages is related to MessageRoutingMode and Message Key. Usually, user would want an ordering of Per-key-partition guarantee" so it is not entirely clear to me if I can achieve what I'm looking to do.
----