You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/04/17 09:11:04 UTC
Slack digest for #general - 2019-04-17
2019-04-16 10:17:15 UTC - stefan: @stefan has joined the channel
----
2019-04-16 10:45:54 UTC - Samuel Sun: Hi all, for pulsar connector , we already have a lot of implementation for different protocol, I wonder if we can have a plugin to make source , pulsar topic/queue, sink together, works like a pipeline thing, this plugin dose not focus the source or target protocol, but can integrate kinds of source and targets implementation only by simple config.
----
2019-04-16 10:47:48 UTC - Ali Ahmed: @Samuel Sun are you about a e2e pipeline ?
----
2019-04-16 13:04:28 UTC - Samuel Sun: @Ali Ahmed not sure what you mean by e2e, what I mean is we could this pipeline plugin config to compose a workflow of data process, like
```
source(any protocol) + queue(non-persistent) + sink(any protocol), what this connector focus is only source push + queue + sink pull without any specialized protocol
```
----
2019-04-16 13:08:29 UTC - Samuel Sun: if it makes senses, I could try to make one, it may focus on make a easier config to help engineer compose different source/target without care about queue implement, just focus workflow of business.
----
2019-04-16 13:10:17 UTC - Alexandre DUVAL: Hi, after cleaning my bookies using metaformat and bookieformat (from our discussion with @David Kjerrumgaard), I have the following issues:
```11:23:36.203 [bookkeeper-ml-workers-OrderedExecutor-3-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [yo/functions/persistent/assignments] Failed to open ledger 1153: Error while recovering ledger
11:23:36.203 [bookkeeper-ml-workers-OrderedExecutor-3-0] WARN org.apache.pulsar.broker.service.BrokerService - Failed to create topic <persistent://yo/functions/assignments>
org.apache.bookkeeper.mledger.ManagedLedgerException: Error while recovering ledger
11:23:36.203 [bookkeeper-ml-workers-OrderedExecutor-3-0] ERROR org.apache.pulsar.broker.service.ServerCnx - [/192.168.10.10:44740] Failed to create topic <persistent://yo/functions/assignments>
java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException$PersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Error while recovering ledger```
Or maybe it's due to errors on cluster metadata initialization
```13:06:12.488 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1014dd3a59a0007
Exception in thread "main" org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /namespace
at org.apache.zookeeper.KeeperException.create(KeeperException.java:122)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792)
at org.apache.pulsar.PulsarClusterMetadataSetup.main(PulsarClusterMetadataSetup.java:178)```
I think it's the same problem
----
2019-04-16 13:11:35 UTC - Alexandre DUVAL: Something relative to ZK, and I'm open to reset and clean my zk nodes
----
2019-04-16 13:39:34 UTC - Sijie Guo: +1 that’s a very useful feature. :+1:
----
2019-04-16 15:35:28 UTC - Devin G. Bost: Hi @Jerry Peng, yes, that's the fully qualified name.
I was able to get functions to successfully create by providing the path to the jar as the second parameter, like this. However, I'm getting some odd results for sources and sinks.
----
2019-04-16 15:42:55 UTC - Sébastien de Melo: Hello guys, I have just pulled the pulsar-all:2.3.1 Docker image and it seems that is has not been updated.
The script /pulsar/bin/apply-config-from-env.py does not contain the last changes, whereas it does in pulsar:2.3.1.
----
2019-04-16 15:47:06 UTC - Sébastien de Melo: By the way, it would be nice to be able to quickly see which version is running
----
2019-04-16 15:49:49 UTC - Matteo Merli: @Sébastien de Melo Let me check the image
----
2019-04-16 15:50:06 UTC - Matteo Merli: The broker will report the version in the logs when it starts
----
2019-04-16 15:50:28 UTC - Sébastien de Melo: OK good to know, thank you!
----
2019-04-16 15:58:49 UTC - Sébastien de Melo: I confirm, the brokers deployed with pulsar-all image report "[main] INFO org.apache.pulsar.broker.PulsarService - Starting Pulsar Broker service; version: '2.3.0.0'" (and '2.3.1' with pulsar image)
----
2019-04-16 16:11:23 UTC - Devin G. Bost: For example, for a sink, I'm getting:
"org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException: HTTP 500 Request failed."
----
2019-04-16 16:12:32 UTC - Devin G. Bost: For a different sink, I'm getting:
"org.apache.pulsar.client.admin.PulsarAdminException: /tmp/pulsar-nar/functions6069595533781956123.tmp-unpacked/META-INF/MANIFEST.MF (No such file or directory)"
I'm not sure where it's getting that directory from.
----
2019-04-16 16:15:02 UTC - Devin G. Bost: If I pass null as the second parameter for sinks, I still get a 500 error for the first sink. For a different sink, I get:
"Sink Package is not provided", which is more expected.
----
2019-04-16 16:31:06 UTC - Devin G. Bost: I'm wondering if it's possible for the sink creation to fail if the configs are pointing to a host that doesn't exist.
----
2019-04-16 16:31:29 UTC - Manoj Manickath: @Manoj Manickath has joined the channel
----
2019-04-16 16:35:17 UTC - Matteo Merli: Yes, I saw the problem. Rebuilding all the images now
----
2019-04-16 16:35:21 UTC - Matteo Merli: Thanks for reporting this!
----
2019-04-16 17:56:36 UTC - Matteo Merli: @Sébastien de Melo Updated the images
+1 : Chris Bartholomew
----
2019-04-16 18:29:34 UTC - Jerry Peng: @Devin G. Bost I am confused at the code you posted before. Are you trying to create or update a function or source?
----
2019-04-16 18:30:31 UTC - Jerry Peng: to create sources and sinks, the admin API for those are different than the one for functions i.e admin.source().create() and admin.since().create()
----
2019-04-16 18:33:51 UTC - Jerry Peng: the sink config you listed above seems correct
----
2019-04-16 18:35:33 UTC - Jerry Peng: @Devin G. Bost what was different between the sinks in which you got a 500 error back and “Sink Package is not provided”?
----
2019-04-16 19:30:15 UTC - Devin G. Bost: @Jerry Peng I apologize for the confusion. I'm upserting functions, sinks, and sources.
For sources, I resolved one of the issues by setting the destinationTopicName as the topicName in the SourceConfig constructor.
After that, I restarted my pulsar standalone docker container, and I stopped seeing the 500 error.
Now, I'm seeing one issue that happens with only some of the sinks and sources, and I'm seeing a different issue that happens with functions at a later point in the bulk upsert operation.
----
2019-04-16 19:30:39 UTC - Devin G. Bost: The remaining issue with some of the sinks and sources is that I'm getting an error like this:
`org.apache.pulsar.client.admin.PulsarAdminException: /tmp/pulsar-nar/functions4700231300249613534.tmp-unpacked/META-INF/MANIFEST.MF (No such file or directory)`
----
2019-04-16 19:31:26 UTC - Devin G. Bost: Here's more of the stack trace:
```
org.apache.pulsar.client.admin.PulsarAdminException: /tmp/pulsar-nar/functions4700231300249613534.tmp-unpacked/META-INF/MANIFEST.MF (No such file or directory)
at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:181)
at org.apache.pulsar.client.admin.internal.SinkImpl.createSink(SinkImpl.java:125)
at com.overstock.deployment.manifest.Sink.upsert(Sink.java:58)
```
----
2019-04-16 19:31:50 UTC - Devin G. Bost: Here's another one:
```
org.apache.pulsar.client.admin.PulsarAdminException: /tmp/pulsar-nar/functions1872994445727810475.tmp-unpacked/META-INF/MANIFEST.MF (No such file or directory)
at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:181)
at org.apache.pulsar.client.admin.internal.SourceImpl.createSource(SourceImpl.java:125)
at com.overstock.deployment.manifest.Source.upsert(Source.java:53)
```
----
2019-04-16 19:32:12 UTC - Devin G. Bost: I'm investigating what's different for these cases because I don't get the error for all of the sinks and sources.
----
2019-04-16 19:33:57 UTC - Devin G. Bost: @Jerry Peng
It appears that the issue _only_ occurs with Kafka sinks and Kafka sources. It also appears that I'm getting the issue on _every_ Kafka sink and _every_ Kafka source.
----
2019-04-16 19:41:52 UTC - Jerry Peng: @Devin G. Bost how are you creating the kafka sources/sinks. Are you submitting the NAR file or are you submitting as a built-in connector?
----
2019-04-16 19:44:34 UTC - Jerry Peng: and which Kafka source/sink are you using?
----
2019-04-16 21:59:12 UTC - Jacob O'Farrell: @Sijie Guo If I don't really care about the data in topics, would I essentially be able to just do a rolling restart of the bookies with the new disk configurations?
----
2019-04-16 22:20:28 UTC - Anthony Corletti: @Anthony Corletti has joined the channel
----
2019-04-16 23:27:01 UTC - Devin G. Bost: Here's a screenshot of how I'm calling the Java Admin API to create/update one of the problematic Kafka sinks. You can see the values being passed in the sinkConfig object.
----
2019-04-16 23:27:11 UTC - Devin G. Bost: @Jerry Peng Does that answer your question?
----
2019-04-16 23:27:51 UTC - Devin G. Bost: Seems to be working now.
----
2019-04-16 23:28:09 UTC - Jerry Peng: @Devin G. Bost so you are submitting your own implementation of the Kafka sink?
----
2019-04-16 23:29:12 UTC - Jerry Peng: I am asking because you are submitting your own JAR
----
2019-04-16 23:29:48 UTC - Devin G. Bost: I'm probably setting the values of the sinkConfig in a misleading way because we're supposed to be using the built-in one. Should I try removing the Jar parameter and see if it works?
----
2019-04-16 23:38:26 UTC - Jerry Peng: You can either submit the connectors in via built-in way if you have built-in connector configured on the broker/worker or submit the connector NAR file. We do also support submitting uber JARs for connectors as well
----
2019-04-16 23:38:36 UTC - Jerry Peng: but I would recommend built-in or NAR
----
2019-04-16 23:39:16 UTC - Jerry Peng: you can download all the NARs for connectors here:
<http://pulsar.apache.org/en/download/>
----
2019-04-17 00:37:38 UTC - Devin G. Bost: Thanks for the info. I'll need to look into this further.
----
2019-04-17 00:42:40 UTC - Devin G. Bost: If I want to use the built-in one, what do I need to pass to the Java Admin API to use it? Is that something that is set on the SourceConfig or SinkConfig object?
----
2019-04-17 01:42:13 UTC - Sijie Guo: I am not sure how k8 handles the new disk configuration change. Does it delete the volume and create a new one? Or does it just expand the size.
If it is the later, you are good.
If it is the former, there will be a problem - bookie has a strict cookie validation mechanism to ensure data consistency. If a bookie’s disk is removed without properly decommission, the bookie will fail the cookie validation. Since the cookie stored locally is removed when the disk was removed, but the cookie stored in zk is still there.
So a better approach will be creating a bookie job with a new disk configuration with a new job name, and deleting the old bookie job.
----
2019-04-17 01:43:28 UTC - Jacob O'Farrell: Sorry - I'm not sure I understand what you mean by "bookie job"
----
2019-04-17 01:46:43 UTC - Sijie Guo: Ah, I mean a new bookkeeper service with a new service name, and a new bookkeeper StatefulSet with a new name. The idea is to make the new bookie have a new hostname with a new disk configuration.
----
2019-04-17 01:48:31 UTC - Jacob O'Farrell: What/where would I need to update to make everything else aware of the new bookies?
----
2019-04-17 01:55:09 UTC - Jacob O'Farrell: P.s. thank you for your help
----
2019-04-17 01:56:48 UTC - Sijie Guo: Just make sure you are using the same zookeeper servers. That’s all the steps you need. The idea is to make sure the bookies are started with new configuration and new bookie id (e.g new bookie hostname in your k8).
----
2019-04-17 05:50:08 UTC - Jacob O'Farrell: Hey all, wondering if anyone has any experience decommissioning bookies - I believe I've successfully begun the process, however the count of ledgers that need to be re-replicated doesn't appear to changing... Is this expected behaviour? <https://gyazo.com/3874dd3dc51659f5cf757d166f028b5a>
----
2019-04-17 08:49:42 UTC - Marc Le Labourier: We are still seeing Starting Pulsar Broker service; version: ‘2.3.0.0’ with apachepulsar/pulsar-all:2.3.1.
Might be that kubernetes did not pull the latest images …
----
2019-04-17 08:58:49 UTC - Marc Le Labourier: It seems to be good now. Thanks @Matteo Merli, @Devin G. Bost.
----