You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/01/31 09:11:04 UTC
Slack digest for #general - 2020-01-31
2020-01-30 09:14:17 UTC - Hiroyuki Yamada: @Hiroyuki Yamada has joined the channel
----
2020-01-30 11:29:54 UTC - Matt Collinge: @Matt Collinge has joined the channel
----
2020-01-30 12:55:18 UTC - Alex Yaroslavsky: Thanks a lot!
----
2020-01-30 12:56:27 UTC - Alex Yaroslavsky: Cool, thanks! We are considering OAuth2 for security.
----
2020-01-30 13:09:57 UTC - Alexandre DUVAL: Will do.
----
2020-01-30 13:20:36 UTC - Roman Popenov: My worker function that is running in the pod still has issues and doesn’t seem to be registering messages that are pushed to the topic
----
2020-01-30 13:23:03 UTC - Roman Popenov: ```root@pulsar-mini-roman-bastion-6568485f5b-9fpdj:/pulsar# bin/pulsar-admin topics stats <persistent://forcepoint-tenant/test-ns/demo-topic-sysmon>
{
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"averageMsgSize" : 0.0,
"storageSize" : 0,
"backlogSize" : 0,
"publishers" : [ {
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"averageMsgSize" : 0.0,
"producerId" : 0,
"metadata" : {
"instance_id" : "0",
"application" : "pulsar-source",
"id" : "forcepoint-tenant/test-ns/s3"
},
"address" : "/10.0.0.254:41086",
"producerName" : "pulsar-mini-roman-1-0",
"connectedSince" : "2020-01-30T13:16:53.382Z",
"clientVersion" : "2.5.0"
} ],
"subscriptions" : {
"my-function-subscription" : {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"msgBacklog" : 0,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"type" : "Shared",
"msgRateExpired" : 0.0,
"lastExpireTimestamp" : 0,
"consumers" : [ {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"consumerName" : "9537e",
"availablePermits" : 1000,
"unackedMessages" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"metadata" : {
"instance_id" : "0",
"application" : "pulsar-function",
"id" : "forcepoint-tenant/test-ns/etl-function"
},
"address" : "/10.0.1.124:45440",
"connectedSince" : "2020-01-30T13:17:23.857Z",
"clientVersion" : "2.5.0"
} ],
"isReplicated" : false
}
},
"replication" : { },
"deduplicationStatus" : "Disabled",
"bytesInCounter" : 6856177,
"msgInCounter" : 2
}```
----
2020-01-30 13:23:13 UTC - Roman Popenov: It is subscribed to it
----
2020-01-30 13:24:11 UTC - Roman Popenov: ```root@pulsar-mini-roman-bastion-6568485f5b-9fpdj:/pulsar# /pulsar/bin/pulsar-admin functions create --jar /tmp/Pulsar-Demo-SKO-1.0-SNAPSHOT-uberjar.jar --classname com.forcepoint.gemini.pulsar.functions.PulsarSysmonTopicETL --namespace test-ns --tenant forcepoint-tenant --inputs <persistent://forcepoint-tenant/test-ns/demo-topic-sysmon> --output <persistent://forcepoint-tenant/test-ns/demo-topic-sysmon-cim> --subs-name my-function-subscription --cpu 1 --ram 3073741824 --disk 10737418240 --parallelism 1 --name etl-function
"Created successfully"```
It was created and the pod is running:
```pf-forcepoint-tenant-test-ns-etl-function-0 1/1 Running 0 6m55s```
----
2020-01-30 13:25:54 UTC - Roman Popenov: I do see the following in the broker logs:
```13:25:06.393 [bookkeeper-io-14-1] WARN org.apache.bookkeeper.proto.PerChannelBookieClient - Exception caught on:[id: 0x746d95a0, L:/10.0.1.224:40326 ! R:/10.0.0.242:3181] cause:
<http://java.io|java.io>.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.8.0_232]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:1.8.0_232]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.8.0_232]
at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:1.8.0_232]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:1.8.0_232]
at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:408) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:931) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.forceFlush(AbstractNioChannel.java:361) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:694) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
13:25:06.394 [bookkeeper-io-14-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0x746d95a0, L:/10.0.1.224:40326 ! R:/10.0.0.242:3181]
13:25:06.394 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (6, 0): Bookie handle is not available
13:25:06.394 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (6, 1): Bookie handle is not available
13:25:06.396 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.bookkeeper.client.BookieWatcherImpl - replaceBookie for bookie: 10.0.0.242:3181 in ensemble: [10.0.0.242:3181, 10.0.0.224:3181] is not adhering to placement policy and chose 10.0.1.125:3181. excludedBookies [10.0.0.242:3181, 10.0.0.225:3181] and quarantinedBookies []```
----
2020-01-30 13:29:33 UTC - Roman Popenov: And this is the stats of the source:
```13:17:50.659 [pulsar-timer-4-1] INFO com.scurrilous.circe.checksum.Crc32cIntChecksum - SSE4.2 CRC32C provider initialized
13:17:53.066 [pulsar-timer-4-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - [<persistent://forcepoint-tenant/test-ns/demo-topic-sysmon>] [pulsar-mini-roman-1-0] Pending messages: 2 --- Publish throughput: 0.03 msg/s --- 6.90 Mbit/s --- Latency: med: 0.000 ms - 95pct: 0.000 ms - 99pct: 0.000 ms - 99.9pct: 0.000 ms - max: -∞ ms --- Ack received rate: 0.00 ack/s --- Failed messages: 0
13:28:53.081 [pulsar-timer-4-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - [<persistent://forcepoint-tenant/test-ns/demo-topic-sysmon>] [pulsar-mini-roman-1-0] Pending messages: 4 --- Publish throughput: 0.03 msg/s --- 6.90 Mbit/s --- Latency: med: 0.000 ms - 95pct: 0.000 ms - 99pct: 0.000 ms - 99.9pct: 0.000 ms - max: -∞ ms --- Ack received rate: 0.00 ack/s --- Failed messages: 0```
----
2020-01-30 13:29:47 UTC - Roman Popenov: So the messages are pending
----
2020-01-30 13:30:00 UTC - Roman Popenov: Not sure what is going on :sweat_smile:
----
2020-01-30 13:34:08 UTC - Roman Popenov: Oh, this might explain it:
```13:30:42.559 [bookie-io-1-1] ERROR org.apache.bookkeeper.proto.BookieRequestHandler - Unhandled exception occurred in I/O thread or handler
io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 5242880: 5313378 - discarded
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:513) ~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:491) ~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.exceededFrameLength(LengthFieldBasedFrameDecoder.java:378) ~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:421) ~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:334) ~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503) ~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281) ~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]```
----
2020-01-30 13:34:21 UTC - Roman Popenov: `TooLongFrameException`
----
2020-01-30 13:50:22 UTC - Roman Popenov: This wasn’t always the behavior, was it?
----
2020-01-30 14:07:34 UTC - Roman Popenov: Is there a place where it can be configured now?
----
2020-01-30 14:33:58 UTC - Alexandre DUVAL: Hi, topic compaction creates a compacted copy of the current topic? I read that the "history" of a topic is kept even after compaction somewhere IIRC, but the cookbook on compaction doesnt mention it.
----
2020-01-30 16:01:45 UTC - David Kjerrumgaard: @Roman Popenov It looks like there is an open issue on this. <https://github.com/apache/pulsar/issues/3832>
----
2020-01-30 16:04:11 UTC - Roman Popenov: I didn’t setup any TLS
----
2020-01-30 16:06:50 UTC - David Kjerrumgaard: You could use OAuth2 JWT for authentication, but that poses an issue with token expiration and being able to revoke a token that has been compromised, etc. Whereas TLS client certs are easier to revoke.
----
2020-01-30 16:10:07 UTC - Roman Popenov: Also, the URLs are:
```{
"serviceUrl" : "<http://pulsar-mini-roman-broker.pulsar.svc.cluster.local:8080/>",
"brokerServiceUrl" : "<pulsar://pulsar-mini-roman-broker.pulsar.svc.cluster.local:6650/>"
}```
----
2020-01-30 16:32:54 UTC - Roman Popenov: I can confirm this has to do with the message size
----
2020-01-30 16:33:40 UTC - Roman Popenov: The setting for messages larger than 5 mbs is not picked up anymore
----
2020-01-30 16:33:50 UTC - Roman Popenov: Is the setting for messages larger than 5 MBs not picked up anymore?
----
2020-01-30 16:34:04 UTC - David Kjerrumgaard: @Roman Popenov My guess is that in the case where TLS was enabled (on the JIRA), that the encryption caused the message size to exceed the 5MB limit and fail.
----
2020-01-30 16:34:35 UTC - Roman Popenov: I don’t have TLS enabled
----
2020-01-30 16:34:48 UTC - Roman Popenov: nor any auth or encryption
----
2020-01-30 16:43:18 UTC - David Kjerrumgaard: Based on your error message, it was picked up. You just sent a message that exceeded that threshold
----
2020-01-30 16:44:12 UTC - David Kjerrumgaard: Adjusted frame length exceeds 5242880 <---- Looks like 5MB
----
2020-01-30 16:44:45 UTC - Alex Yaroslavsky: We already have an automatic JWT renewal mechanism in place.
----
2020-01-30 16:45:18 UTC - Alex Yaroslavsky: And it seems that Pulsar doesn't recheck CRL after a connection was already established (<https://github.com/apache/pulsar/wiki/PIP-20:-Mechanism-to-revoke-TLS-authentication>)
----
2020-01-30 16:45:47 UTC - Roman Popenov: I have previously set
```maxMessageSize: "62428800"```
in broker config, and the borker did pick it up
----
2020-01-30 16:46:01 UTC - Roman Popenov: I’ve also tried 62428800 and 52428800
----
2020-01-30 16:46:30 UTC - Roman Popenov: And it is picked up, but the exception is always the same:
```Adjusted frame length exceeds 5242880: 5313378```
----
2020-01-30 16:46:43 UTC - Roman Popenov: This happens if the message exceeds 5242880 bytes
----
2020-01-30 16:46:54 UTC - Roman Popenov: basically any message over 5 MBs
----
2020-01-30 16:52:08 UTC - Roman Popenov: Could it be related to
----
2020-01-30 16:52:09 UTC - Roman Popenov: <https://github.com/apache/pulsar/issues/4708>
----
2020-01-30 17:00:33 UTC - Roman Popenov: In code, it’s still set as
```bkConf.setNettyMaxFrameSizeBytes(conf.getMaxMessageSize() + Commands.MESSAGE_SIZE_FRAME_PADDING);```
----
2020-01-30 17:02:34 UTC - Alexandre DUVAL: @Matteo Merli
----
2020-01-30 17:16:18 UTC - David Kjerrumgaard: Is it possible to decrease the message size? Chunk the message into smaller pieces?
----
2020-01-30 17:16:42 UTC - Roman Popenov: Yeah, that’s what I will have to do or use 2.4.2, it used to work in that release
----
2020-01-30 17:17:20 UTC - Konstantinos Papalias: Do we still need AggresiveOpts, they have been completely removed from JDK13 now, <https://dzone.com/articles/jdk-13-what-is-aggressiveopts>
```apache-pulsar-2.5.0 % bin/pulsar standalone
Unrecognized VM option 'AggressiveOpts'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.```
----
2020-01-30 17:17:46 UTC - David Kjerrumgaard: Yea, that was my suspicion. This is will need to be changed to support larger message sizes.
----
2020-01-30 17:19:11 UTC - Roman Popenov: I assumed that the message size followed the
```maxMessageSize```
----
2020-01-30 17:19:25 UTC - Roman Popenov: and that the netty size also followed this + padding
----
2020-01-30 17:55:49 UTC - Roman Popenov: It’s really weird, the settings have picked up now, and it does work with larger messages. I will try to repro the issue
thinking_face : David Kjerrumgaard
----
2020-01-30 18:10:38 UTC - Ruslan Sheremet: @Ruslan Sheremet has joined the channel
----
2020-01-30 18:55:12 UTC - Ruslan Sheremet: Hi everyone,
This feature <https://pulsar.apache.org/docs/en/security-jwt/> working in `standalone` mode? I have tried this using `client.conf`, `broker.conf` and `broker.conf` configurations, but auth still not required when I use CLI or a nodejs client. Server run without errors and all working fine, but without enabling Auth via JWT.
`init.sh` used in `docker-compose.yaml` :
```#!/bin/bash
set -e
bin/apply-config-from-env.py conf/broker.conf conf/proxy.conf conf/client.conf \
&& bin/pulsar standalone -a pulsar```
`docker-compose.yaml` :
```version: '3'
services:
pulsar:
image: apachepulsar/pulsar-all
command: bash init.sh
environment:
PULSAR_HOSTNAME: pulsar
PULSAR_PREFIX_webServiceUrl: <http://pulsar:8080/>
PULSAR_PREFIX_brokerServiceUrl: <pulsar://pulsar:6650/>
PULSAR_PREFIX_brokerWebServiceURL: <http://pulsar:8080/>
PULSAR_PREFIX_brokerServiceURL: <pulsar://pulsar:6650/>
PULSAR_PREFIX_advertisedAddress: pulsar
PULSAR_PREFIX_clusterName: standalone
PULSAR_PREFIX_authorizationEnabled: "true"
PULSAR_PREFIX_authenticationEnabled: "true"
PULSAR_PREFIX_authPlugin: org.apache.pulsar.client.impl.auth.AuthenticationToken
PULSAR_PREFIX_authParams: token:eyJhbGciOiJSUzI1NiJ9:eyJzdWIiOiJwdWxzYXIifQ:H0BbDDvywyrept3IouUAzA7-frMBfc-kMa0Bo3JAhiMe-tEhy5QP8Ds3_KVJvyTaE0YI-l5eBK2xucJ3XEdN0x4yMyuXP_VB50mqFoaaLnLL3mpa0qA9kt1sHenmjvijW3yOYs0M8K616Hj_x4hqQq3l-oJ3SKR049LHNm_SvDo478lxo4fx-xqlErRWz7-BJjnYpB0avVEs_-MSaST2PEtAso7hr4XpA84Eh7ydDLCotk3Hm89k9o-e-n6n-Uk-4nwY-PpYyPasHsC7xyqAILvAsLdYeFa3IHDAbnUfvRtixa_phcElM3IU893F-fK1b1po2DKGFAgRW-ie3loLBw
PULSAR_PREFIX_authenticationProviders: org.apache.pulsar.broker.authentication.AuthenticationProviderToken
PULSAR_PREFIX_proxyRoles: proxy,pulsar,pulsar-pulsar
PULSAR_PREFIX_superUserRoles: admin,pulsar,proxy,pulsar-pulsar
PULSAR_PREFIX_tokenPublicKey: file:////pulsar/pulsar-public.key
PULSAR_PREFIX_brokerClientAuthenticationPlugin: org.apache.pulsar.client.impl.auth.AuthenticationToken
PULSAR_PREFIX_brokerClientAuthenticationParameters: token:eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJwdWxzYXIifQ.H0BbDDvywyrept3IouUAzA7-frMBfc-kMa0Bo3JAhiMe-tEhy5QP8Ds3_KVJvyTaE0YI-l5eBK2xucJ3XEdN0x4yMyuXP_VB50mqFoaaLnLL3mpa0qA9kt1sHenmjvijW3yOYs0M8K616Hj_x4hqQq3l-oJ3SKR049LHNm_SvDo478lxo4fx-xqlErRWz7-BJjnYpB0avVEs_-MSaST2PEtAso7hr4XpA84Eh7ydDLCotk3Hm89k9o-e-n6n-Uk-4nwY-PpYyPasHsC7xyqAILvAsLdYeFa3IHDAbnUfvRtixa_phcElM3IU893F-fK1b1po2DKGFAgRW-ie3loLBw
volumes:
- /data/servers/pulsar:/pulsar/data
- /data/servers/pulsar-public.key:/pulsar/pulsar-public.key
- ./pulsar/init.sh:/pulsar/init.sh
ports:
- 6650:6650
- 10180:8080
- 18443:8443
- 6651:6651```
`/data/servers/pulsar-public.key` successfully placed to `/pulsar/*` directory in container and I think should be readable by `file:////pulsar/pulsar-public.key` (absolute path).
Also I have tried to overwrite config files in containers via volumes mount to files, but result the same as with env vars.
----
2020-01-30 18:58:38 UTC - ravi satya durga prasad Yenugula: Thanks the problem was in the document command not sure what the issue one thing is it has `$` I just made it as a single line command and it worked
```docker run -it -p 6650:6650 -p 8080:8080 --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:2.5.0 bin/pulsar standalone```
----
2020-01-30 19:08:25 UTC - David Kjerrumgaard: @Ruslan Sheremet I have it working in a docker image I am using for my book. You can find the code here. <https://github.com/david-streamlio/pulsar-in-action/tree/master/docker-images/pulsar-standalone>
+1 : Ruslan Sheremet
----
2020-01-30 19:09:08 UTC - David Kjerrumgaard: It should give you a solid example of the steps required. Pay particular attention to the configs in the conf/standalone.conf file
----
2020-01-30 19:09:57 UTC - ravi satya durga prasad Yenugula: Hi, I ran the docker command
```docker run -it -p 6650:6650 -p 8080:8080 --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:2.5.0 bin/pulsar standalone```
and trying to access the web UI then I am getting this error
----
2020-01-30 19:11:46 UTC - ravi satya durga prasad Yenugula: Any Idea ?
----
2020-01-30 19:12:19 UTC - Roman Popenov: UI is on port 8080?
----
2020-01-30 19:12:54 UTC - Roman Popenov: I thought that would be the port for admin commands
----
2020-01-30 19:13:04 UTC - Roman Popenov: that you can manage the cluster through REST requests
----
2020-01-30 19:13:39 UTC - ravi satya durga prasad Yenugula: I am new not sure about the tool setup and its functionality
----
2020-01-30 19:14:05 UTC - Roman Popenov: <https://github.com/apache/pulsar-manager> - this manager is running on port 9527
----
2020-01-30 19:14:13 UTC - Roman Popenov: I am not sure if it is running in standalone
----
2020-01-30 19:14:37 UTC - ravi satya durga prasad Yenugula: Doc says `<http://localhost:8080>` so I was try it
----
2020-01-30 19:14:38 UTC - Roman Popenov: ```docker pull apachepulsar/pulsar-manager:v0.1.0
docker run -it -p 9527:9527 -e REDIRECT_HOST=<http://192.168.0.104> -e REDIRECT_PORT=9527 -e DRIVER_CLASS_NAME=org.postgresql.Driver -e URL='jdbc:<postgresql://127.0.0.1:5432/pulsar_manager>' -e USERNAME=pulsar -e PASSWORD=pulsar -e LOG_LEVEL=DEBUG -v $PWD:/data apachepulsar/pulsar-manager:v0.1.0 /bin/sh```
----
2020-01-30 19:16:16 UTC - Ruslan Sheremet: You should use 9527 port in browser
----
2020-01-30 19:16:29 UTC - ravi satya durga prasad Yenugula: I am using `standalone` in the command they mentioned `-p 6650:6650 -p 8080:8080` so its should ne either one. Doc saw 8080 is the http port I am assuming it will have a UI and the 6650 is a pulsar client port
----
2020-01-30 19:17:42 UTC - ravi satya durga prasad Yenugula: `9527` is not working and I am running just the standalone version
----
2020-01-30 19:18:07 UTC - Roman Popenov: No, the port 8080 is used for admin commands and talk to the cluster via rest api
----
2020-01-30 19:18:28 UTC - Roman Popenov: The closest to the UI management of the cluster is <https://pulsar.apache.org/docs/en/administration-pulsar-manager/>
----
2020-01-30 19:18:38 UTC - ravi satya durga prasad Yenugula: Ok
----
2020-01-30 19:18:43 UTC - Ruslan Sheremet: @ravi satya durga prasad Yenugula I mean pulsar-manager:v0.1.0 with apachepulsar/pulsar
----
2020-01-30 19:18:50 UTC - ravi satya durga prasad Yenugula: so it will not have a UI
----
2020-01-30 19:18:57 UTC - Roman Popenov: For the Admin API: <https://pulsar.apache.org/docs/en/admin-api-overview/>
----
2020-01-30 19:19:38 UTC - Ruslan Sheremet: @Roman Popenov Also you should edit nginx config and use domain or IP of your pulsar server
----
2020-01-30 19:20:38 UTC - Ruslan Sheremet: @Roman Popenov
```server {
listen 9527;
server_name 0.0.0.0;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
location /admin {
proxy_pass <http://pulsar:8080>;
}
location /pulsar-admin-doc {
proxy_pass <http://pulsar-admin-swagger:8080>;
}
location /pulsar-manager {
proxy_pass <http://localhost:7750>;
}
location /lookup {
proxy_pass <http://localhost:7750>;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}```
----
2020-01-30 19:24:09 UTC - Roman Popenov: So the suggested command doesn’t work?
----
2020-01-30 19:24:17 UTC - Roman Popenov: <https://pulsar.apache.org/docs/en/administration-pulsar-manager/>
----
2020-01-30 19:24:37 UTC - ravi satya durga prasad Yenugula: I am running the commands
----
2020-01-30 19:26:37 UTC - ravi satya durga prasad Yenugula:
----
2020-01-30 19:26:45 UTC - ravi satya durga prasad Yenugula: docker run is throwing error
----
2020-01-30 19:26:47 UTC - Ruslan Sheremet: @Roman Popenov It is working in case if you are run apachepulsar/pulsar-manager and apachepulsar/pulsar on localhost, but if you want run in VM, this configuration should be changed via nginx config: `/etc/nginx/conf.d/default.conf` (path in Docker container of the apachepulsar/pulsar-manager)
----
2020-01-30 19:28:24 UTC - Roman Popenov: Thanks! :thanks:
----
2020-01-30 19:28:28 UTC - Ruslan Sheremet: @ravi satya durga prasad Yenugula $PWD means current dir
----
2020-01-30 19:28:36 UTC - Roman Popenov: Yes
----
2020-01-30 19:29:18 UTC - Ruslan Sheremet: @Roman Popenov Please note
``` location /pulsar-admin-doc {
proxy_pass <http://pulsar-admin-swagger:8080>;
}```
this is my another config for Swagger API
----
2020-01-30 19:29:45 UTC - Ruslan Sheremet: @Roman Popenov You can exclude this from config
heavy_check_mark : Roman Popenov
----
2020-01-30 19:29:53 UTC - ravi satya durga prasad Yenugula: so show I replace it. In Windows power shell `$PWD` will work
----
2020-01-30 19:42:33 UTC - Ruslan Sheremet: @ravi satya durga prasad Yenugula On 8080 available only Pulsar API with CORS origin. For UI Manager please use Pulsar Manager, default port is `9527`
----
2020-01-30 19:49:01 UTC - ravi satya durga prasad Yenugula: I ran this
```docker run -it -p 9527:9527 -e REDIRECT_HOST=<http://192.168.0.104> -e REDIRECT_PORT=9527 -e DRIVER_CLASS_NAME=org.postgresql.Driver -e URL='jdbc:<postgresql://127.0.0.1:5432/pulsar_manager>' -e USERNAME=pulsar -e PASSWORD=pulsar -e LOG_LEVEL=DEBUG -manager:v0.1.0 /bin/sh```
Error: invalid argument "anager:v0.1.0" for "-m, --memory" flag: invalid size: 'anager:v0.1.0'
then this
```docker run -it -p 9527:9527 -e REDIRECT_HOST=<http://192.168.0.104> -e REDIRECT_PORT=9527 -e DRIVER_CLASS_NAME=org.postgresql.Driver -e URL='jdbc:<postgresql://127.0.0.1:5432/pulsar_manager>' -e USERNAME=pulsar -e PASSWORD=pulsar -e LOG_LEVEL=DEBUG```
error: "docker run" requires at least 1 argument.
See 'docker run --help'.
Usage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]
Run a command in a new container
----
2020-01-30 19:52:38 UTC - Ruslan Sheremet: @ravi satya durga prasad Yenugula You are miss apachepulsar/pulsar-manager:v0.1.0 /bin/sh at the end of run command
----
2020-01-30 19:55:02 UTC - Ruslan Sheremet: @ravi satya durga prasad Yenugula
```docker run -it -p 9527:9527 \
-e REDIRECT_HOST=<http://192.168.0.104> \
-e REDIRECT_PORT=9527 \
-e DRIVER_CLASS_NAME=org.postgresql.Driver \
-e URL='jdbc:<postgresql://127.0.0.1:5432/pulsar_manager>' \
-e USERNAME=pulsar \
-e PASSWORD=pulsar \
-e LOG_LEVEL=DEBUG \
apachepulsar/pulsar-manager:v0.1.0 /bin/sh```
----
2020-01-30 20:02:42 UTC - ravi satya durga prasad Yenugula: This is running
```docker run -it -p 9527:9527 -e REDIRECT_HOST=<http://192.168.0.104> -e REDIRECT_PORT=9527 -e DRIVER_CLASS_NAME=org.postgresql.Driver -e URL='jdbc:<postgresql://127.0.0.1:5432/pulsar_manager>' -e USERNAME=pulsar -e PASSWORD=pulsar -e LOG_LEVEL=DEBUG apachepulsar/pulsar-manager:v0.1.0 /bin/sh```
just changed it to single line
----
2020-01-30 20:24:19 UTC - ravi satya durga prasad Yenugula:
----
2020-01-30 20:24:52 UTC - ravi satya durga prasad Yenugula: When login its is throwing few errors what are those?
```<!doctype html><html lang="en"><head><title>HTTP Status 500 – Internal Server Error</title><style type="text/css">h1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} h2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} h3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} p {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;} a {color:black;} a.name {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 500 – Internal Server Error</h1></body></html>```
----
2020-01-30 20:38:42 UTC - Ruslan Sheremet: @ravi satya durga prasad Yenugula Please follow all changes required for `nginx` config from start of this thread
----
2020-01-30 20:41:50 UTC - Ruslan Sheremet: @ravi satya durga prasad Yenugula You should open
```<http://192.168.0.104:9527>```
----
2020-01-30 20:44:00 UTC - ravi satya durga prasad Yenugula: 9527 is open I am able to access it via localhost
----
2020-01-30 20:47:13 UTC - Ruslan Sheremet: In this case should be REDIRECT_HOST=<http://localhost>
----
2020-01-30 21:00:12 UTC - David Kjerrumgaard: @ravi satya durga prasad Yenugula The image is configured to use TLS for all connections, so you will need to use https:// and port 8443 for the Pulsar REST interface, and pulsar+ssl:// and port 6651 for producers and consumers
----
2020-01-30 21:01:27 UTC - ravi satya durga prasad Yenugula: where should I do it?
----
2020-01-30 21:02:53 UTC - David Kjerrumgaard: I build and tag the image as pia/pulsar-standalone-secure.
----
2020-01-30 21:03:01 UTC - David Kjerrumgaard: Then run it with this command `docker run -id --name pulsar -p:6651:6651 -p 8443:8443 -p 80:80 --volume=${HOME}/exchange:/pulsar/manning/dropbox -t pia/pulsar-standalone-secure:latest`
----
2020-01-30 21:04:29 UTC - David Kjerrumgaard: to get the TLS cert you need to `docker exec -it pulsar bash` and run the following command to "publish" the certs and JWT tokens to your local volume; `/pulsar/manning/securtiy/publish-credentials.sh`
----
2020-01-30 21:05:16 UTC - ravi satya durga prasad Yenugula: Ok I will try and let you know thanks. @Roman Popenov said to use a Manger Image which has UI is it same or some thing different
----
2020-01-30 21:05:44 UTC - ravi satya durga prasad Yenugula: from the image which you are saying
----
2020-01-30 21:06:31 UTC - David Kjerrumgaard: The Pulsar manager is a newer tool, that is NOT included in the Docker image I use.
----
2020-01-30 21:19:26 UTC - David Lanouette: @David Lanouette has joined the channel
----
2020-01-30 21:23:47 UTC - ravi satya durga prasad Yenugula: OMG there are so many ports what do port 80 do?
----
2020-01-30 21:23:59 UTC - ravi satya durga prasad Yenugula: @David Kjerrumgaard
----
2020-01-30 21:24:42 UTC - David Kjerrumgaard: That used to be for the Pulsar Dashboard, but that has been removed in 2.5.0. So you can ignore that for now
----
2020-01-30 21:24:56 UTC - ravi satya durga prasad Yenugula: Ok
----
2020-01-30 21:25:12 UTC - David Kjerrumgaard: The dashboard has been replaced with the manager that @Roman Popenov mentioned
----
2020-01-30 21:25:24 UTC - ravi satya durga prasad Yenugula: Ok
----
2020-01-30 21:25:51 UTC - ravi satya durga prasad Yenugula: I am running it but I see a issue
----
2020-01-30 21:25:55 UTC - David Kjerrumgaard: I am working on a docker-compose now that will support both the secured pulsar image, and the new dashboard component.
----
2020-01-30 21:25:58 UTC - ravi satya durga prasad Yenugula: not sure what I need to do
----
2020-01-30 21:25:59 UTC - David Kjerrumgaard: what is the issue?
----
2020-01-30 21:26:05 UTC - ravi satya durga prasad Yenugula: I will add you to the thread
----
2020-01-30 21:26:08 UTC - David Kjerrumgaard: ok
----
2020-01-30 21:26:29 UTC - ravi satya durga prasad Yenugula: @David Kjerrumgaard please check my above screenshot
----
2020-01-30 21:30:49 UTC - David Kjerrumgaard: You need to change the REDIRECT_HOST in the docker command you used to launch the manager.
----
2020-01-30 21:31:02 UTC - David Kjerrumgaard: to use localhost as suggested.
----
2020-01-30 21:32:14 UTC - ravi satya durga prasad Yenugula: I ran this
```docker run -it -p 9527:9527 -e REDIRECT_HOST=<http://192.168.0.104> -e REDIRECT_PORT=9527 -e DRIVER_CLASS_NAME=org.postgresql.Driver -e URL='jdbc:<postgresql://127.0.0.1:5432/pulsar_manager>' -e USERNAME=pulsar -e PASSWORD=pulsar -e LOG_LEVEL=DEBUG apachepulsar/pulsar-manager:v0.1.0 /bin/sh```
I am using that param
----
2020-01-30 21:32:57 UTC - Roman Popenov: ```REDIRECT_HOST=<http://localhost>```
or 127.0.0.1
----
2020-01-30 21:33:06 UTC - ravi satya durga prasad Yenugula: ok
----
2020-01-30 21:33:47 UTC - ravi satya durga prasad Yenugula: ok
----
2020-01-30 21:39:02 UTC - ravi satya durga prasad Yenugula: Server is up but getting same issue some http error
----
2020-01-30 21:39:13 UTC - Joshua Dunham: Hi Everyone, How can I configure Pulsar to utilize a 'remote' BookKeeper cluster?
+1 : Ryan
----
2020-01-30 21:39:17 UTC - ravi satya durga prasad Yenugula:
----
2020-01-30 21:39:45 UTC - ravi satya durga prasad Yenugula: ```docker run -it -p 9527:9527 -e REDIRECT_HOST=<http://localhost> -e REDIRECT_PORT=9527 -e DRIVER_CLASS_NAME=org.postgresql.Driver -e URL='jdbc:<postgresql://127.0.0.1:5432/pulsar_manager>' -e USERNAME=pulsar -e PASSWORD=pulsar -e LOG_LEVEL=DEBUG apachepulsar/pulsar-manager:v0.1.0 /bin/sh```
changed to localhost every 127...1
still I am getting that http error
----
2020-01-30 21:52:32 UTC - David Kjerrumgaard: The issue appears to be related to the fact that you have specified a Postgres DB that is supposed to be running on localhost but isn't
----
2020-01-30 21:53:00 UTC - ravi satya durga prasad Yenugula: so what should I do now?
----
2020-01-30 21:58:55 UTC - Roman Popenov: <https://github.com/apache/pulsar-manager> - have you tried following the steps in the readme?
----
2020-01-30 22:57:34 UTC - rwaweber: @rwaweber has joined the channel
----
2020-01-30 23:16:15 UTC - David Kjerrumgaard: I haven't gotten this to work yet. Now I am going to have to play with this a bit
----
2020-01-30 23:17:37 UTC - ravi satya durga prasad Yenugula: Please also let me know your findings
----
2020-01-30 23:23:51 UTC - ravi satya durga prasad Yenugula: I tried the environment endpoint as per the gif but I am getting response 404
----
2020-01-30 23:24:57 UTC - ravi satya durga prasad Yenugula: More over the server is keep on throwing the error
```
<!doctype html><html lang="en"><head><title>HTTP Status 500 – Internal Server Error</title><style type="text/css">h1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} h2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} h3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} p {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;} a {color:black;} a.name {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 500 – Internal Server Error</h1></body></html>```
for the most of the clicks :disappointed:
----
2020-01-30 23:26:15 UTC - ravi satya durga prasad Yenugula: The setup is not friendly :disappointed:
----
2020-01-30 23:26:47 UTC - Eugen: Yes, the compacted log will be written to another, internal topic. The consumer can decide which one to consume from by configuring `ConsumerBuilder.readCompacted(bool)`
----
2020-01-30 23:27:16 UTC - Ruslan Sheremet: @ravi satya durga prasad Yenugula Do you run pulsar itself?
----
2020-01-30 23:27:33 UTC - ravi satya durga prasad Yenugula: No
----
2020-01-30 23:34:00 UTC - David Kjerrumgaard: So I have it working now using Docker compose. Here is the relevant file
----
2020-01-30 23:34:07 UTC - David Kjerrumgaard:
----
2020-01-30 23:35:04 UTC - David Kjerrumgaard: Save the above as `docker-compose.yml` and just run `docker-compose up` from the same directory where you save the file.
----
2020-01-30 23:35:11 UTC - ravi satya durga prasad Yenugula: youare using standalone what is the difference between standalone and the master?
----
2020-01-30 23:35:53 UTC - David Kjerrumgaard: you should be able to go to <http://localhost:9527> and manage your Pulsar cluster
----
2020-01-30 23:36:44 UTC - David Kjerrumgaard: The standalone image I am using isn't secured like mine docker image is. So the manager can interact with it.
----
2020-01-30 23:37:14 UTC - David Kjerrumgaard: ^^ This is a limitation with the manager, AFAIK
----
2020-01-30 23:37:36 UTC - ravi satya durga prasad Yenugula: So I have to run both the manger and the standalone
----
2020-01-30 23:37:42 UTC - David Kjerrumgaard: yes
----
2020-01-30 23:39:39 UTC - ravi satya durga prasad Yenugula: ok what about the config that between the master and the standalone box? is it preconfigured in your image
----
2020-01-30 23:46:04 UTC - David Kjerrumgaard: yes
----
2020-01-30 23:46:36 UTC - David Kjerrumgaard: You will have to Add the environment on the <http://localhost:9527/#/environments> link
----
2020-01-30 23:46:57 UTC - David Kjerrumgaard: just enter a name and give a URL of "<http://pulsar:8080>"
----
2020-01-30 23:48:36 UTC - Alexandre DUVAL: Okay, cool!
----
2020-01-30 23:48:38 UTC - Alexandre DUVAL: Thanks
----
2020-01-30 23:54:38 UTC - Sijie Guo: what do you mean “remote”? an existing bookkeeper cluster?
----
2020-01-30 23:56:00 UTC - Roman Popenov: @David Kjerrumgaard doing god’s work :smile:
rolling_on_the_floor_laughing : David Kjerrumgaard
----
2020-01-31 00:15:02 UTC - Eugen: Anyone gotten this, trying to compact a topic? Google hasn't;
```apache-pulsar-2.5.0$ bin/pulsar-admin topics compaction-status <persistent://public/default/my-topic>
Error in compaction
null
Reason: Error compacting: java.util.concurrent.TimeoutException: Timeout```
----
2020-01-31 00:17:11 UTC - David Kjerrumgaard: @Eugen Are there any errors in the broker log for that time period?
----
2020-01-31 00:29:04 UTC - Eugen: I just triggered compaction again
```$ bin/pulsar-admin topics compact <persistent://public/default/my-topic>```
And there were no ERR logs, only one perhaps unrelated WARN. Here are excerpts starting from the first log of the compaction, including the WARN log and all mentions of timeouts:
----
2020-01-31 00:29:04 UTC - Eugen: ```09:23:21.846 [pulsar-30-4] INFO org.apache.pulsar.broker.namespace.OwnershipCache - Trying to acquire ownership of public/default/0x40000000_0x80000000
09:23:21.860 [pulsar-ordered-OrderedExecutor-7-0-EventThread] INFO org.apache.pulsar.broker.namespace.OwnershipCache - Successfully acquired ownership of /namespace/public/default/0x40000000_0x80000000
09:23:21.860 [pulsar-30-11] INFO org.apache.pulsar.broker.PulsarService - Loading all topics on bundle: public/default/0x40000000_0x80000000
09:23:21.862 [pulsar-ordered-OrderedExecutor-5-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger public/default/persistent/my-topic
09:23:21.864 [bookkeeper-ml-workers-OrderedExecutor-2-0] WARN org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [127.0.0.1:3181] is not adhering to Placement Policy. quarantinedBookies: []
09:23:21.885 [pulsar-ordered-OrderedExecutor-7-0-EventThread] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [127.0.0.1:3181] for ledger: 1532
09:23:21.885 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/my-topic] Created ledger 1532
09:23:21.894 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/default/persistent/my-topic] Recovering from bookkeeper ledger cursor: test-my-topic-2
09:23:21.894 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/default/persistent/my-topic] Recovering from bookkeeper ledger cursor: test-1580373893803
09:23:21.894 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/default/persistent/my-topic] Recovering from bookkeeper ledger cursor: test-1580374741644
09:23:21.894 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/default/persistent/my-topic] Recovering from bookkeeper ledger cursor: test-1580374704326
09:23:21.894 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/default/persistent/my-topic] Recovering from bookkeeper ledger cursor: __compaction
09:23:21.894 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/default/persistent/my-topic] Recovering from bookkeeper ledger cursor: test-1580374820773
...
09:23:21.896 [bookkeeper-ml-workers-OrderedExecutor-2-0] WARN org.apache.pulsar.broker.service.AbstractTopic - [<persistent://public/default/my-topic>] Error getting policies java.util.concurrent.CompletableFuture cannot be cast to org.apache.pulsar.common.policies.data.Policies and publish throttling will be disabled
09:23:21.896 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO org.apache.pulsar.broker.service.AbstractTopic - Disabling publish throttling for <persistent://public/default/my-topic>
09:23:21.897 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [<persistent://public/default/my-topic>] There are no replicated subscriptions on the topic
09:23:21.897 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO org.apache.pulsar.broker.service.BrokerService - Created topic <persistent://public/default/my-topic> - dedup is disabled
09:23:21.897 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO org.apache.pulsar.broker.PulsarService - Loaded 1 topics on public/default/0x40000000_0x80000000 -- time taken: 0.036 seconds
09:23:21.908 [pulsar-web-57-7] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer status recorder with config: {
"topicNames" : [ "<persistent://public/default/my-topic>" ],
"topicsPattern" : null,
"subscriptionName" : "__compaction",
"subscriptionType" : "Exclusive",
"receiverQueueSize" : 1000,
"acknowledgementsGroupTimeMicros" : 100000,
"negativeAckRedeliveryDelayMicros" : 60000000,
"maxTotalReceiverQueueSizeAcrossPartitions" : 50000,
"consumerName" : null,
"ackTimeoutMillis" : 0,
"tickDurationMillis" : 1000,
"priorityLevel" : 0,
"cryptoFailureAction" : "FAIL",
"properties" : { },
"readCompacted" : true,
"subscriptionInitialPosition" : "Latest",
"patternAutoDiscoveryPeriod" : 1,
"regexSubscriptionMode" : "PersistentOnly",
"deadLetterPolicy" : null,
"autoUpdatePartitions" : true,
"replicateSubscriptionState" : false,
"resetIncludeHead" : false,
"keySharedPolicy" : null
}
...
09:23:22.006 [BookKeeperClientWorker-OrderedExecutor-5-0] INFO org.apache.bookkeeper.mledger.impl.MetaStoreImplZookeeper - [public/default/persistent/my-topic] [__compaction] Updating cursor info ledgerId=1533 mark-delete=10:-1
09:23:22.025 [pulsar-timer-87-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [<persistent://public/default/my-topic>] [__compaction] Reconnecting after timeout
...
09:23:22.362 [pulsar-client-io-85-1] INFO org.apache.pulsar.compaction.TwoPhaseCompactor - Commencing phase one of compaction for <persistent://public/default/my-topic>, reading to 10:3793:-1
09:23:23.594 [SessionTracker] INFO org.apache.zookeeper.server.ZooKeeperServer - Expiring session 0x10537cac3d3000b, timeout of 30000ms exceeded```
----
2020-01-31 00:32:20 UTC - Eugen: btw is normal that pulsar only logs to stdout when starting it like `bin/pulsar standalone`? It does create a `logs/pulsar-standalone.log` file, but it does not write to it.
----
2020-01-31 00:34:24 UTC - David Kjerrumgaard: ^^^ Yes that is the expected logging behavior. You can override it with the log4j settings
----
2020-01-31 00:34:50 UTC - Eugen: Ok so it's not an issue - never mind!
----
2020-01-31 00:34:50 UTC - Sijie Guo: bin/pulsar script starts component in foreground. so it logs to stdout. because that will be the way how pulsar component be running in a docker container.
the file create is probably related so log4j logger related to function worker.
+1 : Eugen
----
2020-01-31 00:36:10 UTC - David Kjerrumgaard: Nope, just a minor annoyance. The compact error appears to be related to the ZK connection timing out
----
2020-01-31 00:37:49 UTC - Eugen: The compaction timeout occurred only a few seconds after triggering it though, so the 30sec ZK timeout is at least not synchronous to the compaction call
----
2020-01-31 00:39:35 UTC - Alexandre DUVAL: Got exactly the same issue by our side
----
2020-01-31 00:40:07 UTC - David Kjerrumgaard: This looks like an issue. Can one of you please file an issue on GitHub for this?
----
2020-01-31 00:40:24 UTC - Eugen: will do
+1 : David Kjerrumgaard, Alexandre DUVAL, Roman Popenov
----
2020-01-31 01:01:03 UTC - Eugen: I created this issue: <https://github.com/apache/pulsar/issues/6173>
----
2020-01-31 02:04:21 UTC - Eugen: Has anyone tried to use Pulsar to build a system that when a client starts consuming from a topic, it will want to get a snapshot containing the latest value for every key, and then all real-time messages coming in thereafter. Log compaction will only get you halfway there, because it is asynchronous, so clients will receive duplicate keys for the not-yet-compacted part.
----
2020-01-31 02:11:13 UTC - Eugen: Here's how it could perhaps be implemented:
1. have a Pulsar Function that consumes from a (not compacted) topic and continuously updates the latest snapshot in a BookKeeper table, affixing the SequenceId to that table
2. consumers will consume from the topic using InitialPosition `Latest`, taking note of the first SequenceId they receive
3. consumers will get a snapshot from the BookKeeper table with a SequenceId >= that of the first message received via the consumer and merge the snapshot with the messages consumed in real-time
I have not worked with the BookKeeper table service, so I don't know if this is feasible or not. Any ideas?
----
2020-01-31 02:19:13 UTC - Eugen: There are at least 2 ways to do the merging, depending of whether you need to have strict ordering of messages from different keys or not (I would expect order within a single key to always be honored)
----
2020-01-31 03:21:47 UTC - Sijie Guo: The abstraction of this problem is for Pulsar to provide a materialized table view of an “event stream”.
The topic compaction approach is half-through, since it only creates a point-in-time snapshot. To do one step future, we can just implement a “compact/merge-at-read” approach. i.e when processing the compacted view and the uncompacted events, merge them at read time.
The approach you are proposing here will be an approach that is more like a “merge-at-write” approach. as you are materializing the view at the write time to the bookkeeper table.
Both approaches have pros and cons. It depends on what kind of queries you want to support on this “materialized” view.
----
2020-01-31 04:00:52 UTC - Eugen: It seems I have not expressed this clear enough. My approach was definitely meant as a merge-at-read approach, and what I called "snapshot", is what you call materialized table view.
----
2020-01-31 05:45:08 UTC - Sijie Guo: ah okay. then we are on the same page.
----
2020-01-31 05:50:01 UTC - Eugen: by "materialized table view of an event stream", are you referring to the BookKeeper table service?
----
2020-01-31 06:12:57 UTC - Sijie Guo: @Eugen I was referring to a general concept. I didn’t refer to a specific implementation. the main question would be what are the expected operations over this materialized table. is it scan or point lookup? because that can result in different design choices.
----
2020-01-31 06:14:05 UTC - Eugen: it would be scan, as I want to get all key/values
----
2020-01-31 06:15:08 UTC - Eugen: however, it does not technically need to be a real "snapshot" (at one point in time), as long as I have the SequenceId for every key/value pair that I'm scanning
----
2020-01-31 06:19:17 UTC - Ravi Shah: Hi Team, Can i install pulsar c++ client on Alpine?
----
2020-01-31 06:26:52 UTC - Sijie Guo: If it is a scan, you don’t need to materialize to bookkeeper table. since you have a compacted topic (up to given messageId), you can provide a consistent view by reading the compacted topic and merging it with the uncompacted data.
----
2020-01-31 06:33:03 UTC - Eugen: Hm, I'm still debating whether or not to use compaction for this. Will need to play with it to get a feel for it.
----
2020-01-31 06:39:47 UTC - Sijie Guo: my feeling is if your operation is to get all the kvs at given point of time. compaction is probably an efficient way.
but yet. streaming the events to build a table can be an alternative way
----
2020-01-31 07:01:29 UTC - Eugen: Thanks for your input! Will check it out, and if we will end up using Pulsar, I will share my experience here in a couple of weeks to a few months
+1 : Sijie Guo
----
2020-01-31 07:06:26 UTC - Sijie Guo: feel free to ping me if you need any help
heart : Eugen
----
2020-01-31 07:12:28 UTC - Ali Ahmed: @Ravi Shah yes it can be done needs some specific setup I don’t think it’s documented yet
----
2020-01-31 07:20:09 UTC - Ravi Shah: @Ali Ahmed Do you have any link to guide which i can refer or any sort of documentation?
----
2020-01-31 07:20:56 UTC - Ali Ahmed: I can find some time over the weekend to compile it, don’t have anything to share right now
----
2020-01-31 07:27:06 UTC - Ravi Shah: That will be helpful if you can compile it provide. By any chance you know when it will be documented?
----
2020-01-31 07:29:16 UTC - Ali Ahmed: will have to find some time hopefully have something to share next week
----