You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/08/21 09:11:04 UTC
Slack digest for #general - 2019-08-21
2019-08-20 11:48:48 UTC - Alexandre DUVAL: @Sijie Guo @Jerry Peng I have very strange behavior after deleting schema :open_mouth:
----
2019-08-20 11:49:34 UTC - Alexandre DUVAL: @Sijie Guo @Jerry Peng more readable here
----
2019-08-20 11:49:45 UTC - Alexandre DUVAL: got strange behavior on schemas API
----
2019-08-20 11:51:25 UTC - Alexandre DUVAL: listings schemas of a topic return 404, get the current schema return the schema
----
2019-08-20 11:51:34 UTC - Alexandre DUVAL: delete returns Ok, but it's not deleted
----
2019-08-20 12:13:45 UTC - Sijie Guo: > listings schemas of a topic return 404,
`Problem accessing /admin/v2/schemas/yo/applicationslogs/null-addon_298bd840-6935-4fee-9de2-f0632172e0d7-raw/schemas`
If you are using 2.4.0, there is no methods to list all the schemas. The API is only added recently. 404 means that endpoint doesn’t exist. Not the schema doesn’t exist.
----
2019-08-20 12:15:26 UTC - Sijie Guo: > delete returns Ok, but it’s not deleted
```
yo-pulsar-c1-n7 /pulsar # curl -X DELETE -H "Authorization: Bearer <ADMIN_TOKEN>" -k -i '<https://c1-pulsar-yo-customers.services.yo.com:2000/admin/v2/schemas/yo/applicationslogs/null-addon_298bd840-6935-4fee-9de2-f0632172e0d7-raw/schema>' -L
HTTP/1.1 200 OK
Date: Tue, 20 Aug 2019 11:44:35 GMT
Date: Tue, 20 Aug 2019 11:44:35 GMT
Content-Type: application/json
Server: Jetty(9.4.12.v20180830)
Content-Length: 13
{"version":1}%
```
from the delete response, it seems that you deleted the version 1 of the schema. but the version 2 schema still exists. that explains why you can still get the schema.
----
2019-08-20 12:15:44 UTC - Alexandre DUVAL: how to delete the version 2?
----
2019-08-20 12:17:03 UTC - Alexandre DUVAL: @Sijie Guo
----
2019-08-20 12:19:56 UTC - Sijie Guo: what version are you using? it seems to be a bug to me.
----
2019-08-20 12:20:05 UTC - Alexandre DUVAL: checking
----
2019-08-20 12:20:55 UTC - Alexandre DUVAL: 2.4.0 on brokers
----
2019-08-20 12:23:46 UTC - Sijie Guo: have you deleted schema on same topic before?
----
2019-08-20 12:33:39 UTC - Alexandre DUVAL: yes
----
2019-08-20 12:34:25 UTC - Alexandre DUVAL: you meant schema version? I don't know, here i just want to clean all schemas for topics
----
2019-08-20 12:35:15 UTC - Alexandre DUVAL: the point is, i tried avroschema on pulsar function context.publish and it failed, i pushed rowbytes with "avro" schema as last values so schemas look strange
----
2019-08-20 12:35:41 UTC - Alexandre DUVAL: thanks to Jerry, i fixed this, but a lot of of topics were created on these bad schemas
----
2019-08-20 12:36:09 UTC - Alexandre DUVAL: so now, i added try catch on producer creation in my pulsar function to handle invalid schema and delete them directly then retry
----
2019-08-20 12:36:18 UTC - Alexandre DUVAL: and here i am to try debug this via curl
----
2019-08-20 12:41:04 UTC - Alexandre DUVAL: don't know how to clean all for a topic given
----
2019-08-20 12:41:44 UTC - Alexandre DUVAL: and as I said with @Jerry Peng, we are not able to add all schemas for all topics fetch because the schemalocator depends on schemaId :confused:.
----
2019-08-20 12:43:59 UTC - Sijie Guo: If that’s the case, you hit a known issue about handling deleted schema. the fix is in 2.4.1 and master. @jia zhai is working on releasing 2.4.1. It should be out soon.
----
2019-08-20 12:47:30 UTC - Sijie Guo: you need a fix in 2.4.1 to handle deleted schemas
----
2019-08-20 12:49:15 UTC - Alexandre DUVAL: from which commit can I build it ?
----
2019-08-20 12:49:41 UTC - Alexandre DUVAL: or maybe latest build of master is ok?
----
2019-08-20 12:59:16 UTC - Sijie Guo: latest master is okay.
----
2019-08-20 13:04:31 UTC - Alexandre DUVAL: i need to update all component or only brokers are enough?
----
2019-08-20 13:12:44 UTC - Sijie Guo: only brokers
----
2019-08-20 13:14:08 UTC - Alexandre DUVAL: ok, done and fixed, thanks ! :slightly_smiling_face:
----
2019-08-20 13:18:03 UTC - Sijie Guo: cool
----
2019-08-20 13:45:33 UTC - Petr Tichy: @Petr Tichy has joined the channel
----
2019-08-20 15:04:35 UTC - Tarek Shaar: I have created many topics for my testing and I would like to delete some topics. Is there a way to delete a group of topics together? It seems I only allowed to delete one topic at a time.
----
2019-08-20 15:35:34 UTC - Alexandre DUVAL: How is defined the RAM allowed to pulsar function java-instance.jar?
----
2019-08-20 16:10:00 UTC - Alexandre DUVAL: i mean how ram is managed between pulsar-function-worker and java-instances (functions)
----
2019-08-20 17:14:59 UTC - Matteo Merli: You can also avoid to explicitely delete topics. Just close all producers/consumers and unsubscribe from the topics. The topic will be automatically deleted
----
2019-08-20 17:25:38 UTC - Jerry Peng: @Alexandre DUVAL in process runtime, the ram set for a function will be the -Xmx setting for the java instance
----
2019-08-20 17:26:26 UTC - Jerry Peng: For thread runtime, the ram setting is ignored
----
2019-08-20 17:26:35 UTC - Alexandre DUVAL: We can define them on java-instance, there is a conf field? because currently they use the PULSAR_MEM env I think.
----
2019-08-20 17:27:34 UTC - Jerry Peng: PULSAR_MEM is for the xmx setting for brokers and pulsar function workers (if run separately)
----
2019-08-20 17:28:25 UTC - Jerry Peng: When a function worker starts a java instance (separate process), the -Xmx is set to the corresponding value that the user has set for the ram of the function
----
2019-08-20 17:28:40 UTC - Jerry Peng: from the CLI, to set the ram “--ram”
----
2019-08-20 17:41:08 UTC - Alexandre DUVAL: Ok, IIRC this option is documented as only working for Docker.
----
2019-08-20 17:41:42 UTC - Jerry Peng: oh where is that documentation?
----
2019-08-20 17:42:31 UTC - Alexandre DUVAL: Oh it has been fixed.
----
2019-08-20 17:42:45 UTC - Alexandre DUVAL: ``` --ram
The ram in bytes that need to be allocated per function
instance(applicable only to process/docker runtime)
```
----
2019-08-20 17:42:54 UTC - Alexandre DUVAL: pulsar-admin cli
----
2019-08-20 17:43:17 UTC - Alexandre DUVAL: IIRC it was only about docker, but mb fixed or missread. thanks :slightly_smiling_face:.
----
2019-08-20 19:35:48 UTC - Alexandre DUVAL: Doo you know how to make avro schema generation "generic" without java-class fields specified?
----
2019-08-20 20:44:08 UTC - Addison Higham: general question: anyone implemented some of the pulasr-admin in golang? can't find anything
----
2019-08-20 20:53:01 UTC - Jacob Fugal: I have :wink: but I know that doesn't answer why you're asking
----
2019-08-20 20:53:46 UTC - Jacob Fugal: (we work together, and he wants, rightly, to make my continued work on it unnecessary)
----
2019-08-20 22:52:38 UTC - Matteo Merli: We started a proof of concept several months ago, but that was left there (for lack of time..) . it was certainly faster than the Java based one, though significant work was needed to cover all the commands for pulsar-admin
----
2019-08-21 06:22:45 UTC - Jacob O'Farrell: Hi all - looking to potentially migrate from Pulsar 2.3.0 to Pulsar 2.4.0. I noticed that the release notes don't mention any migration notes, breaking changes or similar. Is there anything I should be weary of?
----
2019-08-21 06:29:43 UTC - tuteng: You can refer to this <http://pulsar.apache.org/docs/en/next/administration-upgrade/>
----
2019-08-21 08:11:48 UTC - Timmisetty K: @Timmisetty K has joined the channel
----