You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/02/09 23:11:00 UTC

[GitHub] [pulsar] yabinmeng opened a new issue #9549: Pulsar topic compaction through "pulsar compact-topic" is not working

yabinmeng opened a new issue #9549:
URL: https://github.com/apache/pulsar/issues/9549


   **Describe the bug**
   I'm testing pulsar topic compaction using both `pulsar-admin topics compact` and `pulsar compact-topic` commands. 
   * `pulsar-admin topics compact` command works fine
   * `pulsar compact-topic` command fails right away 
   
   I tested with versions 2.6.3 and 2.7.0. Both fail, but with different errors.
   
   **2.6.3** (**NOTE**: my pulsar cluster doesn't have any security feature, including TLS, enabled at all)
   ```
   5:40.800 [pulsar-client-io-11-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x9b33e1cf, L:/10.101.32.196:52186 - R:ip-10-101-32-196.srv101.dsinternal.org/10.101.32.196:6651]] Connected to server
   20:25:40.824 [pulsar-client-io-11-1] WARN  org.apache.pulsar.client.impl.ClientCnx - Error during handshake
   java.nio.channels.ClosedChannelException: null
           at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1076) [io.netty-netty-handler-4.1.48.Final.jar:4.1.48.Final]
           at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) 
   ... ... 
   ```
   
   **2.7.0**
   ```
   Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.NullPointerException
   	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
   	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
   	at org.apache.pulsar.compaction.CompactorTool.main(CompactorTool.java:108)
   Caused by: java.lang.NullPointerException
   	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:878)
   	at org.apache.bookkeeper.zookeeper.ZooKeeperClient$Builder.build(ZooKeeperClient.java:225)
   	at org.apache.pulsar.zookeeper.ZookeeperBkClientFactoryImpl.lambda$0(ZookeeperBkClientFactoryImpl.java:55)
   	at org.apache.bookkeeper.util.SafeRunnable$2.safeRun(SafeRunnable.java:73)
   	at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
   	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
   	at java.lang.Thread.run(Thread.java:748)
   ```
   
   **To Reproduce**
   Steps to reproduce the behavior:
   1.  Create a producer and publish some messages to a topic 
   2.  Run `pulsar compact-topic --topic <topic_name>` 
   
   **Expected behavior**
   A clear and concise description of what you expected to happen.
   
   **Screenshots**
   If applicable, add screenshots to help explain your problem.
   
   **Desktop (please complete the following information):**
    - OS: [e.g. iOS]
   
   **Additional context**
   Add any other context about the problem here.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] WJL3333 commented on issue #9549: Pulsar topic compaction through "pulsar compact-topic" is not working

Posted by GitBox <gi...@apache.org>.

WJL3333 commented on issue #9549:
URL: https://github.com/apache/pulsar/issues/9549#issuecomment-777606794


   @yabinmeng do you config any tlsport like `brokerServicePortTls`. According to code logic if config `brokerServicePortTls` will try to connect server in tls. see `org.apache.pulsar.compaction.CompactorTool` and check again.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] yabinmeng commented on issue #9549: Pulsar topic compaction through "pulsar compact-topic" is not working

Posted by GitBox <gi...@apache.org>.

yabinmeng commented on issue #9549:
URL: https://github.com/apache/pulsar/issues/9549#issuecomment-777584512


   @WJL3333 you're right. I was executing command `pulsar compact-topic --topic <topic_name>` under my home directory. Based on your suggestion, I ran the command under pulsar installation home directory as `bin/pulsar compact-topic --topic <topic_name>`.
   
   The NPE error does go away; then it comes to the error of SSL hand-shake failure, as experienced in Pulsar 2.6.3. But as I mentioned, the cluster doesn't have any security feature enabled. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] WJL3333 commented on issue #9549: Pulsar topic compaction through "pulsar compact-topic" is not working

Posted by GitBox <gi...@apache.org>.

WJL3333 commented on issue #9549:
URL: https://github.com/apache/pulsar/issues/9549#issuecomment-777287962


   @yabinmeng hi, can you check where you run the `pulsar compact-topic --topic <topic_name>`.
   if you don't specify config file path. the logic right now will read `${pwd}/conf/broker.conf` as configuration file.
   which may cause read the wrong file and `zookeeperServers` is blank.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] yabinmeng commented on issue #9549: Pulsar topic compaction through "pulsar compact-topic" is not working

Posted by GitBox <gi...@apache.org>.

yabinmeng commented on issue #9549:
URL: https://github.com/apache/pulsar/issues/9549#issuecomment-776803224


   @WJL3333 here is the part of the broker.conf (for 2.7 cluster). Almost all settings are default except the following ones. zookeeperServers config is set and it is as below (note the cluster is a 3-node cluster)
   ```
   # Zookeeper quorum connection string
   zookeeperServers=<node1_ip>:2181,<node2_ip>:2181,<node3_ip>:2181
   
   # Configuration Store connection string
   configurationStoreServers=<node1_ip>:2181,<node2_ip>:2181,<node3_ip>:2181
   
   # Name of the cluster to which this broker belongs to
   clusterName=mypulsar_cluster 
   ```
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] jiazhai closed issue #9549: Pulsar topic compaction through "pulsar compact-topic" is not working

Posted by GitBox <gi...@apache.org>.

jiazhai closed issue #9549:
URL: https://github.com/apache/pulsar/issues/9549


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] yabinmeng edited a comment on issue #9549: Pulsar topic compaction through "pulsar compact-topic" is not working

Posted by GitBox <gi...@apache.org>.

yabinmeng edited a comment on issue #9549:
URL: https://github.com/apache/pulsar/issues/9549#issuecomment-778290565

@WJL3333 Yes, I have `brokerServicePortTls` set with port 6551. But `**tlsEnabled**` is set as false and other TLS related settings (e.g. `tlsCertificateFilePath` etc.) are empty
```
brokerServicePort=6650
brokerServicePortTls=6651
tlsEnabled=false
tlsCertificateFilePath=
```

Now I do see in broker.conf comment it says that `tlsEnabled` is deprecated (and the code indeed does the only check of the existence of tls port). For "deprecated" feature/setting, my understanding is that the old setting should still be supported until it is officially removed. Before then, both the old setting and new setting should be respected. When both settings are present, the old one should take precedence.

Also, after a quick search of `tlsEnabled` in the code base and looks like it is still respected in a few other places (e.g. PulsarChannelInitializer.java); but not in other places. This is a bit confusing.

The official document (2.7.0) is also still based on the check of `tlsEnabled`:
https://pulsar.apache.org/docs/en/security-tls-transport/#broker-configuration

Looks like we can do some improvement here

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] WJL3333 commented on issue #9549: Pulsar topic compaction through "pulsar compact-topic" is not working

Posted by GitBox <gi...@apache.org>.

WJL3333 commented on issue #9549:
URL: https://github.com/apache/pulsar/issues/9549#issuecomment-776514743


   can you upload your `broker.conf`? i think if `zookeeperServers` config is set the 2.7 NPE will gone.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] yabinmeng commented on issue #9549: Pulsar topic compaction through "pulsar compact-topic" is not working

Posted by GitBox <gi...@apache.org>.

yabinmeng commented on issue #9549:
URL: https://github.com/apache/pulsar/issues/9549#issuecomment-778290565

The official document (2.7.0) is also still based on the check of `tlsEnabled`:
https://pulsar.apache.org/docs/en/security-tls-transport/#broker-configuration

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] yabinmeng edited a comment on issue #9549: Pulsar topic compaction through "pulsar compact-topic" is not working

Posted by GitBox <gi...@apache.org>.

yabinmeng edited a comment on issue #9549:
URL: https://github.com/apache/pulsar/issues/9549#issuecomment-778290565

@WJL3333 Yes, I have `brokerServicePortTls` set with port 6651. But `**tlsEnabled**` is set as false and other TLS related settings (e.g. `tlsCertificateFilePath` etc.) are empty
```
brokerServicePort=6650
brokerServicePortTls=6651
tlsEnabled=false
tlsCertificateFilePath=
```

The official document (2.7.0) is also still based on the check of `tlsEnabled`:
https://pulsar.apache.org/docs/en/security-tls-transport/#broker-configuration

Looks like we can do some improvement here

For queries about this service, please contact Infrastructure at:
users@infra.apache.org