Slack digest for #general - 2018-01-12

[Apache Pulsar Slack <ap...@gmail.com>]

2018-01-11 22:36:03 UTC - Tiago Caxias: @Tiago Caxias has joined the channel
----
2018-01-12 00:19:20 UTC - Jaebin Yoon: I'm trying to bring up pulsar in my company's environment where I created a wrapper for pulsar broker and bookie. Basically the wrapper sets up company environment and starts the broker or bookie server. In the runtime, I'm getting protobuf-java jar library issue since the wrapper requires 3.2.0 while pulsar is using 2.4.1 (I'm using pulsar-1.21.0). 
The following is the error message I'm getting with protobuf-java-3.2.0.jar in runtime.
```java.lang.NoSuchMethodError: com.google.protobuf.TextFormat.printToString(Lcom/google/protobuf/Message;)Ljava/lang/String;
        at org.apache.bookkeeper.bookie.Cookie.toString(Cookie.java:121)
        at org.apache.bookkeeper.bookie.Cookie.writeToDirectory(Cookie.java:175)
        at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:359)
        at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:481)```

Is there any reason that pulsar broker and bookeeper need to stay with protobuf-java-2.4.1 ? Any issue with moving to 3.x ?
----
2018-01-12 02:36:38 UTC - Jaebin Yoon: I just got the yahoo forked Bookkeeper and regenerated code from protobuf definitions with protoc (v3.5) and built the bookkeeper-server jar. It seems it works with that. I noticed that apache bookkeeper has already moved to protobuf 3.4.0
----
2018-01-12 04:16:24 UTC - Jaebin Yoon: I tried to do the same thing for pulsar broker but it seems the pulsar broker is using the modified version of protobuf code generator. Hmm not sure the patch would work with v3.5.0
----
2018-01-12 06:44:07 UTC - Sijie Guo: @Jaebin Yoon can you try pulsar-broker-shaded?
----
2018-01-12 07:14:01 UTC - Jaebin Yoon: oh what is it? it seems it's just a packaging. What I'm doing is that I have my own java main to hook up in-house environment and have dependency to pulsar-broker and instantiate PulsarService object from that main.
----
2018-01-12 07:20:01 UTC - Jaebin Yoon: The crux of the problem is that pulsar is using protobuf 2.x and our environment is using protobuf 3.x.  I'm trying to apply the pulsar protobuf patch to 3.x and generate code. That seems to be the only way if I want to keep my environment and pulsar in the same JVM.
----
2018-01-12 08:49:18 UTC - Matteo Merli: Hi @Jaebin Yoon, I think the problem would be there even using protobuf 3.x, because the code generated is typically not compatible with the runtime lib of a different minor version.
----
2018-01-12 08:50:12 UTC - Matteo Merli: The only option is indeed to shade protobuf (and other potentially conflicting dependencies) when embedding the broker into an existing Java process
----
2018-01-12 08:51:44 UTC - Matteo Merli: initially we were just shading the client lib, though recently @Nicolas Ha added support to have the shaded broker artifact, but that will be included in 1.22 release
----
2018-01-12 08:53:51 UTC - Matteo Merli: if you want to have the “shaded broker jar” without waiting for 1.22 (or using master), you could also create a module in your project that depends on pulsar broker and then applies the shading. You just need a `pom.xml` like in <https://github.com/apache/incubator-pulsar/tree/master/pulsar-broker-shaded>
----
2018-01-12 08:57:58 UTC - Matteo Merli: &gt; Is there any reason that pulsar broker and bookeeper need to stay with protobuf-java-2.4.1 ? Any issue with moving to 3.x ?

In any case, yes, we’ll move to 3.x soon, after we do the switch from BookKeeper-4.3-Yahoo to BookKeeper-4.7. The patch itself is very easy to port to the new protobuf version (The code is just adding hooks so that we can avoid all allocation when serializing/deserializing protobuf objects)
----
2018-01-12 09:04:02 UTC - Matteo Merli: @Jaebin Yoon I actually stand corrected: `pulsar-broker-shaded` was already included in 1.21 release. <https://mvnrepository.com/artifact/org.apache.pulsar/pulsar-broker-shaded>
----
2018-01-12 09:53:31 UTC - Nicolas Ha: Yes - I had the exact same problem :smile: Using the published version of `pulsar-broker-shaded` works for me with my current protobuf dependency (same JVM as well).

Note: I would still welcome `SNAPSHOT` versions at some point - this would make it easier to experiment with not shading Netty for instance.
----
2018-01-12 17:15:27 UTC - Jaebin Yoon: @Matteo Merli @Sijie Guo oh this is nice. I didn't know about maven-shade-plugin. I'll try that. Thanks a lot!!
----