You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/05/14 09:11:04 UTC
Slack digest for #general - 2019-05-14
2019-05-13 09:16:33 UTC - Alexandre DUVAL: Hi, when you apply a schema to a topic, then you add a function using this topic as input. The process function signature looks like (<SchemaType> input, Context context)?
----
2019-05-13 09:25:35 UTC - Alexandre DUVAL: I see options ``` --custom-schema-inputs
The map of input topics to Schema class names (as a JSON string)
--custom-serde-inputs
The map of input topics to SerDe class names (as a JSON string) ``` on pulsar-admin functions create, but do you have a quick example of the full usage?
----
2019-05-13 09:44:14 UTC - bhagesharora: Hello everyone, How consumer.seek(msg_id) method is working in case of retention policy . If we set the time limit 5m or size flag 5M ??
----
2019-05-13 12:00:32 UTC - Justin:
----
2019-05-13 12:06:41 UTC - Justin: This is my domain class:
public class PulsarMqData<T> implements Serializable {
private static final long serialVersionUID = -1;
private T data;
private int count = 0;
private long ct = 0L;
private long rt = 0L;
private long pct = 0L;
private long prt = 0L;
//getter setter
}
----
2019-05-13 12:35:40 UTC - Alexandre DUVAL: My topic schema is `{"version":2,"type":"STRING","timestamp":0,"data":"","properties":{}}`, but I got ```12:30:07.109 [pulsar-io-21-4] WARN org.apache.pulsar.broker.service.ServerCnx - [/192.168.10.15:36928] Got exception BufferUnderflowException : null
java.nio.BufferUnderflowException: null
at java.nio.Buffer.nextGetIndex(Buffer.java:506) ~[?:1.8.0_192]
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:412) ~[?:1.8.0_192]
at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.versionFromBytes(BookkeeperSchemaStorage.java:169) ~[org.apache.pulsar-pulsar-broker-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
at org.apache.pulsar.broker.service.schema.SchemaRegistryServiceImpl.versionFromBytes(SchemaRegistryServiceImpl.java:128) ~[org.apache.pulsar-pulsar-broker-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
at org.apache.pulsar.broker.service.ServerCnx.handleGetSchema(ServerCnx.java:1328) ~[org.apache.pulsar-pulsar-broker-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
at org.apache.pulsar.common.api.PulsarDecoder.channelRead(PulsarDecoder.java:299) ~[org.apache.pulsar-pulsar-common-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:433) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:330) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
``` on brokers node when I start my functions. It's all about String. Consume works well with random pulsar client.
----
2019-05-13 12:47:36 UTC - Alexandre DUVAL: It's not taking the highest version by default?
----
2019-05-13 15:23:10 UTC - Devin G. Bost: Any updates on this? We're scheduled to go to beta in two days, and we need 2.3.2 for our incremental update code.
----
2019-05-13 15:33:26 UTC - Nathan Linebarger: @Nathan Linebarger has joined the channel
----
2019-05-13 16:58:06 UTC - Matteo Merli: @bhagesharora When trying to seek to a message id that was already deleted, it will get you to the earliest available message
----
2019-05-13 19:06:26 UTC - Thor Sigurjonsson: @Devin G. Bost Permissions on /data/provisioning causing bookie to not come up?
----
2019-05-13 19:07:07 UTC - Devin G. Bost: Permissions are `rwxrwxrw-` with me as the owner.
----
2019-05-13 19:10:58 UTC - Thor Sigurjonsson: does /data exist in the container?
----
2019-05-13 19:12:48 UTC - Thor Sigurjonsson: You doing this on macos?
----
2019-05-13 19:19:49 UTC - Devin G. Bost: Yes and yes.
----
2019-05-13 20:05:53 UTC - Devin G. Bost: I accidentally deleted my thread. Today is just one of those days.
----
2019-05-13 20:09:47 UTC - Devin G. Bost: I was missing the `-v $PWD/data:/pulsar/data` part of the docker command. That's what the problem was.
This works:
```
docker run -it \
-p 6650:6650 \
-p 8080:8080 \
-v $PWD/data:/pulsar/data \
-v /data/provisioning:/data/provisioning \
apachepulsar/pulsar-standalone \
bin/pulsar standalone
```
----
2019-05-14 00:10:04 UTC - Devin G. Bost: What does it mean when I get `Reason: Function in trigger function is not ready` when I try to run `bin/pulsar-admin functions trigger` on my Pulsar function?
----
2019-05-14 00:48:07 UTC - Devin G. Bost: I figured out that there was a problem with how it was created.
----
2019-05-14 01:01:13 UTC - Devin G. Bost: I'm attempting to use a Pulsar function for continuous deployment to construct other Pulsar functions via the Java Admin API.
When I send it the config data for a Pulsar component that I want it to create through the Java Admin API, I'm getting this error:
```
00:47:00.452 [pulsar-client-io-70-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [<persistent://ops/deployment/incrementalUpdates>] [standalone-0-3] Closed Producer
org.apache.pulsar.client.admin.PulsarAdminException: A MultiException has 2 exceptions. They are:
1. java.lang.NoSuchMethodError: org.apache.pulsar.common.util.ObjectMapperFactory.create()Lorg/apache/pulsar/shade/com/fasterxml/jackson/databind/ObjectMapper;
2. java.lang.IllegalStateException: Unable to perform operation: create on org.apache.pulsar.client.admin.internal.JacksonConfigurator
at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:187)
at org.apache.pulsar.client.admin.internal.TenantsImpl.getTenants(TenantsImpl.java:50)
at com.overstock.dataeng.pulsar.deployment.manifest.Tenant.existsInPulsar(Tenant.java:119)
. . .
```
----
2019-05-14 01:13:12 UTC - Devin G. Bost: I checked the versions of the dependencies in my function, and they match Pulsar's dependency versions. (I'm using Pulsar 2.3.1.)
----
2019-05-14 01:18:25 UTC - Devin G. Bost: It seems surprising that the `getTenants` call would result in this exception.
----
2019-05-14 01:21:26 UTC - Jerry Peng: @Devin G. Bost are you calling getTenant within your function?
----
2019-05-14 01:21:31 UTC - Devin G. Bost: Yes.
----
2019-05-14 01:21:54 UTC - Devin G. Bost: @Jerry Peng It's that line in the stacktrace:
`com.overstock.dataeng.pulsar.deployment.manifest.Tenant.existsInPulsar(Tenant.java:119)`
----
2019-05-14 01:22:30 UTC - Jerry Peng: @Devin G. Bost you included pulsar-client-admin in the pom as a dependency of the function?
----
2019-05-14 01:22:48 UTC - Jerry Peng: also are you building an uber jar for the function?
----
2019-05-14 01:23:43 UTC - Jerry Peng: based on the stacktrace, a probable cause is that the pular-client-admin JAR/Classes is not included in the function JAR
----
2019-05-14 01:24:09 UTC - Devin G. Bost: I'll check on that. Yes, it's a rather large Jar.
----
2019-05-14 01:25:00 UTC - Jerry Peng: @Devin G. Bost you can inspect the contents of your JAR and make sure class “org/apache/pulsar/shade/com/fasterxml/jackson/databind/ObjectMapper” is in it.
----
2019-05-14 01:25:03 UTC - Devin G. Bost: This is in my POM:
```
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-client-admin</artifactId>
<version>${pulsar.version}</version>
</dependency>
```
----
2019-05-14 01:25:10 UTC - Devin G. Bost: Sure thing.
----
2019-05-14 01:25:37 UTC - Jerry Peng: Thats good but make sure you are building the function JAR as an uber JAR that includes this dependency
----
2019-05-14 01:26:13 UTC - Devin G. Bost: Like this?
```
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.1.1</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>assemble-all</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
```
----
2019-05-14 01:28:26 UTC - Devin G. Bost: Here's what I found in my Jar in the directory org/apache/pulsar/shade/com/fasterxml/jackson/databind/
----
2019-05-14 01:29:12 UTC - Devin G. Bost: `ObjectMapper.class` is in there.
----
2019-05-14 01:34:29 UTC - Jerry Peng: @Devin G. Bost this is after you unzipped the JAR?
----
2019-05-14 01:34:33 UTC - Devin G. Bost: Yes.
----
2019-05-14 01:36:05 UTC - Jerry Peng: Actually the exception indicates that ```org.apache.pulsar.common.util.ObjectMapperFactory``` class is not found
----
2019-05-14 01:36:15 UTC - Jerry Peng: please check for that class
----
2019-05-14 01:36:38 UTC - Devin G. Bost: Oh, good catch. I wasn't sure how to interpret the exception message.
----
2019-05-14 01:36:42 UTC - Devin G. Bost: I'll check.
----
2019-05-14 01:37:56 UTC - Devin G. Bost: @Jerry Peng Sure enough, `ObjectMapperFactory.class` is in `/org/apache/pulsar/common/util/` in the unpacked jar.
----
2019-05-14 01:41:04 UTC - Devin G. Bost: I just re-built the jar and double-checked, and both of those classes are present.
----
2019-05-14 01:46:52 UTC - Jerry Peng: I think I know the issue. There is probably a class/shading conflict with java-instance.jar. ObjectMapperFactory is included with java-instance.jar but its looking for org/apache/pulsar/functions/runtime/shaded/com/fasterxml/jackson/databind/ObjectMapper. While pulsar-client-admin is looking for another version. I will have to shaded the pulsar-common dependency in java-instance.jar. Users don’t usually use pulsar-admin API inside functions. We can still support it
----
2019-05-14 01:47:28 UTC - Jerry Peng: @Devin G. Bost I will make a PR to fix the shading to make this work
----
2019-05-14 01:48:09 UTC - Devin G. Bost: @Jerry Peng Thank you very much for supporting this use case! This is a key part of our CI/CD pipeline architecture and plans.
----
2019-05-14 01:48:44 UTC - Jerry Peng: @Devin G. Bost np shouldn’t be a difficult fix. I will have a PR up shortly
----
2019-05-14 01:50:36 UTC - Devin G. Bost: Thanks @Jerry Peng! We're actually scheduled to go to beta on Wednesday, so we will eagerly look forward to the fix... :sweat_smile:
----
2019-05-14 01:53:24 UTC - Jerry Peng: @Devin G. Bost as an alternative to consider instead of using the pulsar-admin API you can use a HTTP client and call out the the Pulsar’s REST interface directly which is what the pulsar-admin API is doing underneath anyways.
----
2019-05-14 01:53:45 UTC - Jerry Peng: Obviously that would be more burdensome and you will have to for the http requests
----
2019-05-14 01:54:34 UTC - Devin G. Bost: Thanks for the suggestion. However, at this point, that would be a bit of a re-write, so I think we will need to explore another workaround until you guys are able to ship the fix.
----
2019-05-14 02:04:19 UTC - Devin G. Bost: @Jerry Peng Do you think it would work if we created a consumer with the Pulsar client library?
----
2019-05-14 02:11:49 UTC - Matteo Merli: Yes, if you just want to ensure a subscription is there, you can just subscribe and immediately close the consumer.
----
2019-05-14 02:12:32 UTC - Devin G. Bost: Sounds good. Thanks for the guidance.
----
2019-05-14 04:38:39 UTC - Sanjeev Kulkarni: @Devin G. Bost I would highly recommend against using pulsar admin library inside your function. It’s not really meant to be a general purpose library
----
2019-05-14 04:39:04 UTC - Sanjeev Kulkarni: The idea of using a http interface or some other simpler rest interface is what I would recommend as well
----
2019-05-14 07:20:40 UTC - Jerry Peng: @Devin G. Bost ya I would also recommend that as well
----
2019-05-14 08:47:59 UTC - bhagesharora: Ok.. Thanks for your response, Can you please explain where we need to configure or mention time limit or size limit. even I tried out with pulsar-admin command and changing in broker.conf file. so After the publishing message in consumer acknowledge message is not coming but when I am using reader interface(rewind process) message is still coming.
----