You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/05/03 09:11:02 UTC
Slack digest for #general - 2019-05-03
2019-05-02 10:02:00 UTC - Alexandre DUVAL: There is a way to get exception `10:00:58.222 [function-timer-thread-27-1] ERROR org.apache.pulsar.functions.runtime.RuntimeSpawner - yo/functions/RouteApplicationsAddonsLogs-java.lang.RuntimeException: Function Container is dead with exception.. restarting` or I should add logs and get them from logs topic of the function?
----
2019-05-02 10:20:16 UTC - Alexandre DUVAL: localrun give it, no problem then.
----
2019-05-02 10:29:17 UTC - David Tinker: Upgrading from 2.1.1-incubating to 2.3.1: Is there an upgrade guide somewhere?
----
2019-05-02 10:53:59 UTC - Sijie Guo: I had a PR - <https://github.com/apache/pulsar/pull/2048/files>
but didn’t get a chance to complete it.
<https://github.com/apache/pulsar/blob/a8a1453486de6e94a76284503f80c3148bf819c3/site/docs/latest/admin/Upgrade.md>
it should give you some ideas
----
2019-05-02 11:00:28 UTC - Shivji Kumar Jha: @Sijie Guo I think this should be merged. We have been looking for a good upgrade guide for sometime, this can be very useful :slightly_smiling_face:
----
2019-05-02 11:01:01 UTC - Sijie Guo: yes I will try to get some time to work on it again this week :slightly_smiling_face:
----
2019-05-02 11:01:46 UTC - Shivji Kumar Jha: Awesome :smile:
----
2019-05-02 12:16:06 UTC - Laurent Chriqui: Hello, when publishing messages with the python client in async mode, I have noticed in the callback that all messages ids are (-1,-1,-1,-1) is that normal? (in cases of success or failure)
----
2019-05-02 12:49:13 UTC - stefan: hi guys, i have set up a two clusters geo replicated instance. it works fine when i use command to test and produce/consume directly on the broker, but when using a java software client from a micro-service, i managed to connect to a broker through the internal dns but i end up with that exception
org.apache.pulsar.shade.io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 5242880: 1213486164 - discarded
when trying to create a producer
----
2019-05-02 12:49:43 UTC - stefan: tls is disabled and i am using <pulsar://dns:6650> to access the broker
----
2019-05-02 12:49:50 UTC - stefan: any help appreciated
----
2019-05-02 15:00:35 UTC - Poule: anyone has problem running python3 functions on macOS?
----
2019-05-02 15:01:00 UTC - Poule: all i get when restarting the function is `"UNAVAILABLE: io exception`
----
2019-05-02 15:01:51 UTC - Poule: nothing in /logs/
----
2019-05-02 15:03:30 UTC - Poule: works great with python2
----
2019-05-02 15:04:58 UTC - Poule: `pulsar-admin functions querystate --fqfn public/default/exclamation` gives me `Reason: java.lang.NullPointerException: path is 'null'.`
----
2019-05-02 15:41:14 UTC - Poule: I opened pulsar in Intellij and followed instructions to set up annotations and lombok
----
2019-05-02 15:42:57 UTC - Poule: what do I need to put in the Debug config to start debugging
----
2019-05-02 15:52:54 UTC - Sanjeev Kulkarni: @Poule i’m able to run fnctions in python3 on my mac(10.14.4)
----
2019-05-02 15:54:05 UTC - Poule: @Sanjeev Kulkarni what is the path of your python3 binary
----
2019-05-02 15:54:13 UTC - Poule: and version of python 3 is?
----
2019-05-02 15:54:24 UTC - Sanjeev Kulkarni: which python
/usr/local/bin/python
----
2019-05-02 15:54:33 UTC - Poule: python -V
----
2019-05-02 15:54:36 UTC - Sanjeev Kulkarni: python
Python 3.7.2 (default, Feb 12 2019, 08:15:36)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type “help”, “copyright”, “credits” or “license” for more information.
----
2019-05-02 15:55:26 UTC - Poule: you use pulsar 2.3.1 with pulsar-client==2.3.1 ?
----
2019-05-02 15:58:59 UTC - Sanjeev Kulkarni: this is from master
----
2019-05-02 16:03:20 UTC - AlexStocks: @AlexStocks has joined the channel
----
2019-05-02 16:03:22 UTC - Poule: do you run function with .zip or .py
----
2019-05-02 16:04:01 UTC - Sanjeev Kulkarni: py
----
2019-05-02 16:04:21 UTC - Sanjeev Kulkarni: are you running in standalone/
----
2019-05-02 16:04:25 UTC - Poule: yes
----
2019-05-02 16:04:26 UTC - Sanjeev Kulkarni: do you have any broker logs?
----
2019-05-02 16:04:47 UTC - Poule: nothing in the logs, it must fails before logging
----
2019-05-02 16:05:09 UTC - Sanjeev Kulkarni: i mean not the function logs but broker logs
----
2019-05-02 16:05:24 UTC - Sanjeev Kulkarni: there might be some exception that is occuring while the broker is trying to start the function
----
2019-05-02 16:05:49 UTC - Poule: I tail all files /logs and nothing
----
2019-05-02 16:06:02 UTC - Poule: in
----
2019-05-02 16:06:40 UTC - Poule: usually I see python errors in logs but not in this case
----
2019-05-02 16:11:34 UTC - Sanjeev Kulkarni: when you start standalone, it logs stuff on the stdout. maybe you can see some exceptions there
----
2019-05-02 16:14:49 UTC - Poule: yes `ProcessRuntime - Extracted Process death exception`
----
2019-05-02 16:15:04 UTC - Poule: ` Function Container is dead with exception.. restarting`
----
2019-05-02 16:15:45 UTC - Poule: and in function status I have ` "error" : "UNAVAILABLE: io exception",`
----
2019-05-02 16:16:07 UTC - Poule: at this point the only solution I see is debug in intellij
----
2019-05-02 16:16:29 UTC - Poule: but I don't know how to config the RUN config for debug
----
2019-05-02 16:27:26 UTC - Sanjeev Kulkarni: ok, in the broker messages, you should also see the cmd line that the broker is trying to execute for the process
----
2019-05-02 16:27:42 UTC - Sanjeev Kulkarni: Could you run that same command from a terminal and see where that crashes
----
2019-05-02 16:41:55 UTC - Poule: you mean the python_instance_main.py call
----
2019-05-02 16:44:01 UTC - Poule: `Segmentation fault: 11`
----
2019-05-02 16:48:54 UTC - Sanjeev Kulkarni: what was the command that you executed?
----
2019-05-02 16:54:04 UTC - Poule: ```
python /Users/poule/dev/infrastructure/pulsar/pulsar/apache-pulsar-2.3.1/instances/python-instance/python_instance_main.py --py /tmp/pulsar_functions/public/default/exclamation/0/exclamation.zip --logging_directory /Users/poule/dev/infrastructure/pulsar/pulsar/apache-pulsar-2.3.1/logs/functions --logging_file exclamation --logging_config_file /Users/poule/dev/infrastructure/pulsar/pulsar/apache-pulsar-2.3.1/conf/functions-logging/logging_config.ini --instance_id 0 --function_id 989ea3b7-3c96-4b8a-81c4-b1e1ed92ca3a --function_version 85e19965-a9e3-4e46-ab6c-2268f6313d9d --function_details '{"tenant":"public","namespace":"default","name":"exclamation","className":"exclamation","runtime":"PYTHON","autoAck":true,"parallelism":1,"source":{"inputSpecs":{"<persistent://public/default/backwards>":{}},"cleanupSubscription":true},"sink":{"topic":"<persistent://public/default/forwards>"},"resources":{"cpu":1.0,"ram":"1073741824","disk":"10737418240"}}' --pulsar_serviceurl <pulsar://127.0.0.1:6650> --use_tls false --tls_allow_insecure false --hostname_verification_enabled false --max_buffered_tuples 1024 --port 49382 --metrics_port 49383 --state_storage_serviceurl <bk://127.0.0.1:4181> --expected_healthcheck_interval 30 --secrets_provider secretsprovider.ClearTextSecretsProvider --cluster_name standalone
```
----
2019-05-02 16:54:43 UTC - Poule: it does not SegFault with python2, only with python3
----
2019-05-02 17:11:13 UTC - Sanjeev Kulkarni: can you do pip list and grep for pulsar
----
2019-05-02 17:11:21 UTC - Sanjeev Kulkarni: see which pulsar-client is installed
----
2019-05-02 17:35:00 UTC - Poule: ```
import pulsar
import Function_pb2
import log
import server
import python_instance
import util
```
----
2019-05-02 17:35:26 UTC - Poule: those lines cause the segfault ^^
----
2019-05-02 17:36:21 UTC - Poule: relative imports changed in python3, maybe a python guru can tell us if it's still legal to import this way
----
2019-05-02 17:46:47 UTC - Sanjeev Kulkarni: i dont think relative imports cause segfaults
----
2019-05-02 17:46:57 UTC - Sanjeev Kulkarni: which is the version of your pulsar-client
----
2019-05-02 17:47:22 UTC - Sanjeev Kulkarni: pip show pulsar-client
----
2019-05-02 18:05:59 UTC - Poule: 2.3.1
----
2019-05-02 18:06:30 UTC - Poule: ```
[/u/l/bin]$ pip show pulsar-client
Name: pulsar-client
Version: 2.3.1
Summary: Apache Pulsar Python client library
Home-page: <https://pulsar.apache.org/>
Author: Pulsar Devs
Author-email: <ma...@pulsar.apache.org>
License: Apache License v2.0
Location: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
Requires: six, ratelimit, grpcio, prometheus-client, fastavro, apache-bookkeeper-client, protobuf
Required-by:
```
----
2019-05-02 18:07:39 UTC - Sanjeev Kulkarni: does import pulsar cause a segfault for you?
----
2019-05-02 18:57:36 UTC - Poule: all relative imports
----
2019-05-02 18:59:26 UTC - Poule: and import pulsar too
----
2019-05-02 20:25:09 UTC - Poule: now i am at `python-instance/pulsar/schema/schema.py` where it segfaults at `import _pulsar`
----
2019-05-02 20:27:06 UTC - Poule: i don't know how to go further
----
2019-05-02 20:29:42 UTC - Matt Grosso: Hi, I'm helping my team consider a switch to Pulsar. One potential stumbling block I'm trying to think through is related to how Kafka offsets can be compared with integer `<` whereas MessageId are opaque.
----
2019-05-02 20:30:04 UTC - Matt Grosso: The reason this matters is we are using an EventSourced architecture, meaning in this case that in one particular topic, we send a variety of messages that each have part of the state for objects we can call Foo. One event might denote that some Foo with id 42 has baz="Baahz" while another might set bar="Baahr" on the same object. We have a micro service that relays these events to clients via WebSockets and our clients assemble those events into the complete picture of all the Foo.
----
2019-05-02 20:30:45 UTC - Matt Grosso: We need this WebSocket micro service to have a reasonable startup time and also to provide its clients with all the Events they may be missing. With Kafka we solve this by keeping track of the oldest offset, which works because older offsets are `<` than newer ones. It doesn't look like MessageId can be compared according to the API, so I'm wondering how other Pulsar users have solved this issue.
----
2019-05-02 20:30:56 UTC - Matt Grosso: Are there patterns and anti-patterns specific to Pulsar here? maybe utilizing compaction?
----
2019-05-02 20:41:48 UTC - Poule: you need to compare a msgid with another?
----
2019-05-02 20:42:17 UTC - Matt Grosso: in short, yes.
----
2019-05-02 20:42:48 UTC - Poule: your messages are not in order?
----
2019-05-02 20:43:35 UTC - Matt Grosso: no. clients can store up messages while offline and then submit them.
----
2019-05-02 20:45:05 UTC - Matt Grosso: The correct way to sort the messages is based on message content, so all clients know how to sort old messages. Also, we're looking at using Pulsar partly because of its geo-replication feature, and I assume that a global ordering of messages is impossible if they are submitted by producers local to different clusters within the instance.
----
2019-05-02 20:53:38 UTC - Poule: you don't have a read model?
----
2019-05-02 20:58:05 UTC - Matt Grosso: It's on our clients.
----
2019-05-02 20:58:17 UTC - David Kjerrumgaard: @Matt Grosso Couldn't you achieve the desired results by using message properties in Pulsar?
----
2019-05-02 20:58:49 UTC - Matt Grosso: which are clients of the websocket microservice.
----
2019-05-02 20:59:43 UTC - David Kjerrumgaard: @Matt Grosso You might want to look at the MessageBuilder API, it allows you to attach the EventTime, Sequence ID, and various properties to a message
----
2019-05-02 20:59:46 UTC - David Kjerrumgaard: <http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/MessageBuilder.html>
----
2019-05-02 20:59:52 UTC - Poule: yes i understand but in ES your read model is not regenerated each time the client connects
----
2019-05-02 21:01:47 UTC - Fredrick P Eisele: @David Kjerrumgaard I noticed that MessageBuilder has deprecated all over it <http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/Producer.html#newMessage--> and <http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/TypedMessageBuilder.html> seem to be the replacement.
----
2019-05-02 21:02:01 UTC - Matt Grosso: thankyou, that `setSquenceId` looks shiny.
+1 : David Kjerrumgaard
----
2019-05-02 21:03:21 UTC - David Kjerrumgaard: @Fredrick P Eisele You are correct,. What can I say, I'm old school and wrote a lot of code using that API :smiley:
----
2019-05-02 21:03:58 UTC - David Kjerrumgaard: FWIW, the ability to attach the EventTime, Sequence ID, and various properties to a message exists in the `TypedMessageBuilder` as well :smiley:
+1 : Matt Grosso
----
2019-05-02 21:08:12 UTC - Matt Grosso: right. The clients maintain state across reconnects, but a new client can show up without any state. I'd like to shrink the number of messages a WebSocket micro service has to read in order to "catchup" and be able to service those new blank clients. Suppose a service were to act as read model and maintain the message ids and their composition; if it could compare message ids across objects (but within a partition) then it could periodically emit the oldest, such that when restarted *without any state other than that messageId and access to Pulsar* it could rebuild it's state in a time proportional to currently active messages, rather than proportional to all the events ever.
----
2019-05-02 21:08:30 UTC - Alexandre DUVAL: What's the limit of pulsar functions jar file can be uploaded to functions_worker? I have issues about jersey space limit for an uber jar of 40M.
----
2019-05-02 21:11:19 UTC - Matt Grosso: maybe the disconnect is that we aren't maintaining a separate persistent read model server side.
----
2019-05-02 21:12:16 UTC - Fredrick P Eisele: I am having some confusion over what <http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/MessageId.html#latest> will do.
If I open a reader with it....
```
Reader reader = client.newReader().topic("a-topic").startMessageId(MessageId.latest).create();
```
The documentation says `MessageId.latest : Start reading from the end topic, only getting messages published after the reader was created `
What I want is the last message extant before the reader was created.
Specifically I want the `event_timestamp` of the latest message on a topic.
----
2019-05-02 21:13:36 UTC - David Kjerrumgaard: @Alexandre DUVAL I don't have an exact number to share, but what I do know is that when I need to upload a large uber jar or NAR file, I have to adjust the memory configs in my `pulsar_tools_env.sh` file from the default setting of 128m
----
2019-05-02 21:21:01 UTC - David Kjerrumgaard: @Fredrick P Eisele So you want to read only one previous message and all messages that are published after the reader was created? In order to achieve that, you will need to provide that information to the reader and pass in the particular message id.
----
2019-05-02 21:22:08 UTC - David Kjerrumgaard: @Fredrick P Eisele An alternative to that (if you don't have a way of getting the messageId before hand) is to start at the earliest available and read forward until you get to the end, ignoring all but the last message
----
2019-05-02 21:24:44 UTC - David Kjerrumgaard:
----
2019-05-02 21:24:47 UTC - Fredrick P Eisele: @David Kjerrumgaard I am writing a source connector and some of the messages have previously been loaded. I want to pick up from the last message successfully loaded. I could keep the message id around but it seems silly given that I have it in Pulsar.
----
2019-05-02 21:29:43 UTC - David Kjerrumgaard: Pulsar keeps track of that via the cursor position in a subscription. Does your connector delete the subscription, or is it transient (is spun up and down)?
----
2019-05-02 21:32:36 UTC - Fredrick P Eisele: It is transient.
----
2019-05-02 21:32:55 UTC - David Kjerrumgaard: When you say "connector" I am assuming you are talking about custom connector like descibed here: <http://pulsar.apache.org/docs/en/io-develop/>
----
2019-05-02 21:33:04 UTC - Fredrick P Eisele: That is right.
----
2019-05-02 21:37:06 UTC - David Kjerrumgaard: I wonder if making it stateful would help? You could record the previous message id to the context and on startup, read that information from the context object. I am not quite sure how that work for a connector that gets deleted though
----
2019-05-02 21:37:47 UTC - David Kjerrumgaard: I believe the context object would be deleted when the connector was
----
2019-05-02 21:38:53 UTC - Fredrick P Eisele: I am working a several connectors, most are very custom.
One is to provide a source connector for MongoDB (I have updated the sink connector and will release them together).
+1 : David Kjerrumgaard
----
2019-05-02 21:40:30 UTC - David Kjerrumgaard: The stateful feature would be useful for other scenarios, such as a high-water mark for a JDBC connector that pulls data from a table based on a PK, and needs to record that last PK it processed, etc.
----
2019-05-02 21:42:01 UTC - Fredrick P Eisele: Exactly, I was hoping this was a solved problem and you could point me to an example.
----
2019-05-02 21:42:37 UTC - David Kjerrumgaard: If not, it is a good candidate for a PIP request
----
2019-05-02 21:42:40 UTC - Alexandre DUVAL: Hi @David Kjerrumgaard, which configuration field do you change?
----
2019-05-02 21:42:58 UTC - Alexandre DUVAL: Because it's a 40Mo uber jar, and doesn't fit.
----
2019-05-02 21:43:15 UTC - David Kjerrumgaard: `PULSAR_MEM="-Xmx128m -XX:MaxDirectMemorySize=128m"`
----
2019-05-02 21:43:42 UTC - David Kjerrumgaard: I bump that up to 1GB for both
----
2019-05-02 21:44:09 UTC - Alexandre DUVAL: It's not related? How it should be? I mean it's functions_worker multipartconfig, no?
----
2019-05-02 21:45:28 UTC - Fredrick P Eisele: I was just looking at @Matt Grosso’s recent thread. Is he looking at something similar?
"if it could compare message ids across objects (but within a partition) then it could periodically emit the oldest, such that when restarted *without any state other than that messageId and access to Pulsar* it could rebuild it's state in a time proportional to currently active messages" -- <https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1556828982321600>
----
2019-05-02 21:46:11 UTC - David Kjerrumgaard: The uploading of the NAR file is done via HTTP, so I would get OOMs when trying to upload larger archives
----
2019-05-02 21:47:22 UTC - Alexandre DUVAL: It is not OOM, it's about REST API for uploading jar limit: ```
null
Reason: javax.ws.rs.ProcessingException: Java heap space
```
----
2019-05-02 21:49:40 UTC - David Kjerrumgaard: yep, that is the issue I was referring to....
----
2019-05-02 21:50:18 UTC - Matt Grosso: yeah, The kafka pattern I was hoping to reuse is to be able to compare messageIds and write out the oldest to a separate meta topic. When restarting, read the last message from the meta topic to get the offset I need to start reading from.
----
2019-05-02 21:51:08 UTC - Alexandre DUVAL: The functions_worker node has 7G of heap, the client which is uploading locally has 2G, same issue. (The jar size is 40Mo).
----
2019-05-02 21:51:16 UTC - Matt Grosso: The problem is I can't just restart from the next message *after* I subscribe, I have to rebuild state in the client, but I don't want to go back to the dawn of time to do that, and I'd like to avoid storing state outside Pulsar.
----
2019-05-02 21:52:32 UTC - Alexandre DUVAL: As it's multipart limit on server side (functions_worker api) so not a problem of heap for the java program, but only for multipartconfig no?
----
2019-05-02 21:53:53 UTC - Matt Grosso: @Fredrick P Eisele maybe if the meta topic is compacted frequently then reading from the beginning will be `O(close enough)`
----
2019-05-02 21:54:40 UTC - David Kjerrumgaard: What command are you running when you get this error?
----
2019-05-02 21:55:08 UTC - Alexandre DUVAL: ```bin/pulsar-admin --admin-url <https://192.168.10.16:2001> --auth-params "token:TOKEN" functions create --jar /home/kannar/git/kannar/apache-pulsar-2.3.1/conf/functions/target/pulsar-functions-0.1.0-SNAPSHOT.jar --classname com.yo.pulsar.function.RouteApplicationsAddonsLogs --inputs yo/logs/full --tenant yo --namespace functions --logTopic log_topic
```
----
2019-05-02 21:56:39 UTC - David Kjerrumgaard: and what is your `PULSAR_MEM=` set to in your `pulsar_tools_env.sh` file? 7GB?
----
2019-05-02 21:56:50 UTC - Alexandre DUVAL: 1g for tools
----
2019-05-02 21:56:58 UTC - Alexandre DUVAL: and 7g for nodes
----
2019-05-02 21:59:47 UTC - David Kjerrumgaard: PULSAR_MEM="-Xmx1g -XX:MaxDirectMemorySize=1g" on the node where you are running the `functions create`? That should work
----
2019-05-02 22:00:34 UTC - Alexandre DUVAL: No, i run the crate functions locally, from my laptop. To the admin url of the functions worker.
----
2019-05-02 22:00:41 UTC - Alexandre DUVAL: the first one ig 1g for the client
----
2019-05-02 22:00:47 UTC - Alexandre DUVAL: and thefunctions worker node is 7g
----
2019-05-02 22:03:12 UTC - David Kjerrumgaard: The PULSAR_MEM settings has to increased on the machine where you submit the JAR file from, so in this case your laptop. That is because the uploading of file occurs locally inside the netty server. Try upping the memory on your laptop
----
2019-05-02 22:05:20 UTC - Fredrick P Eisele: @Matt Grosso ...maybe. Also using the `earliest` only gets you the earliest unacknowledged message. Could that help? I will start with the re-reader that @David Kjerrumgaard suggested and see how it performs.
----
2019-05-02 22:12:06 UTC - Matt Grosso: interesting, that's a good point and I'm not sure. I'd have to think that through.
----
2019-05-02 22:26:18 UTC - Alexandre DUVAL: ```kannar@pond ~/git/kannar/apache-pulsar-2.3.1 % export PULSAR_MEM='-Xmx3g -XX:MaxDirectMemorySize=4g'
kannar@pond ~/git/kannar/apache-pulsar-2.3.1 % echo $PULSAR_MEM
-Xmx3g -XX:MaxDirectMemorySize=4g
kannar@pond ~/git/kannar/apache-pulsar-2.3.1 % bin/pulsar-admin --admin-url <https://192.168.10.16:2001> --auth-params "token:TOKEN" functions create --jar /home/kannar/git/kannar/apache-pulsar-2.3.1/conf/functions/target/pulsar-functions-0.1.0-SNAPSHOT.jar --classname com.clevercloud.pulsar.function.RouteApplicationsAddonsLogs --inputs clevercloud/logs/full --tenant clevercloud --namespace functions --logTopic log_topic
null
Reason: javax.ws.rs.ProcessingException
```
----
2019-05-02 22:26:48 UTC - Alexandre DUVAL: I'm pretty sure it's limit on multipart inputstream size server side so on the functions_worker node. Nothing related to pulsar_mem.
----
2019-05-02 22:37:21 UTC - David Kjerrumgaard: The problem with exporting the PULSAR_MEM variable, is that it gets overridden by the value in the `pulsar_tools_env.sh` file
----
2019-05-02 22:38:07 UTC - David Kjerrumgaard: "BINDIR=$(dirname "$0")
export PULSAR_HOME=`cd $BINDIR/..;pwd`
. "$PULSAR_HOME/bin/pulsar-admin-common.sh" <---- the entire content of bin/pulsar-admin
----
2019-05-02 22:38:25 UTC - David Kjerrumgaard: DEFAULT_CLIENT_CONF=$PULSAR_HOME/conf/client.conf
DEFAULT_LOG_CONF=$PULSAR_HOME/conf/log4j2.yaml
if [ -f "$PULSAR_HOME/conf/pulsar_tools_env.sh" ]
then
. "$PULSAR_HOME/conf/pulsar_tools_env.sh"
fi
----
2019-05-02 22:38:37 UTC - David Kjerrumgaard: the first 5 lines of PULSAR_HOME/bin/pulsar-admin-common.sh
----
2019-05-02 22:39:13 UTC - David Kjerrumgaard: # Extra options to be passed to the jvm
PULSAR_MEM="-Xmx128m -XX:MaxDirectMemorySize=128m"
----
2019-05-02 22:39:36 UTC - David Kjerrumgaard: ^^^ Where the value gets overridden inside ../conf/pulsar_tools_env.sh
----
2019-05-02 22:40:06 UTC - David Kjerrumgaard: so please try changing the above file and resubmitting the function
----
2019-05-02 22:43:18 UTC - Alexandre DUVAL: Ohhh, ok. Will retry. Thanks.
----
2019-05-02 22:54:19 UTC - Alexandre DUVAL: Ahah `Reason: javax.ws.rs.ProcessingException: Request timeout to clevercloud-pulsar-c1-n7/192.168.10.16:2001 after 60000 ms`
----
2019-05-02 22:55:08 UTC - David Kjerrumgaard: that is progress.....
----
2019-05-02 22:55:42 UTC - David Kjerrumgaard: looks like your process isn't listing on that port
----
2019-05-02 22:59:14 UTC - Alexandre DUVAL: No, I think the uploads is quite long :stuck_out_tongue:.
----
2019-05-02 23:41:02 UTC - Poule: yes that's why i asked for better state in Pulsar a while back: <https://github.com/apache/pulsar/issues/2489>
----
2019-05-02 23:42:23 UTC - Poule: in Kafka Streams we can query KTable by range if I remember. I'd like to have this in Pulsar
----
2019-05-02 23:43:57 UTC - Poule: Rocksdb is queryable by range so I guess it would be sortable
----
2019-05-02 23:46:47 UTC - Poule: Goal would be to replace Cassandra as read model db
----
2019-05-02 23:51:19 UTC - Matt Grosso: cool. I'll follow that issue. In the meantime I think it's possible to code around the problem using a compacted meta topic that has message_ids, at least for our use case, if not the more general one.
----