You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "BewareMyPower (via GitHub)" <gi...@apache.org> on 2023/04/25 13:56:06 UTC
[GitHub] [pulsar-client-cpp] BewareMyPower opened a new pull request, #257: Support get the SchemaInfo from a topic and the schema version
BewareMyPower opened a new pull request, #257:
URL: https://github.com/apache/pulsar-client-cpp/pull/257
### Motivation
Currently there is no public API to get the SchemaInfo from the broker. However, when a consumer tries to decode a message of Avro schema, the writer schema that was stored in the broker is required. The C++ client is not responsible to decode the message, but the extended library like the Python client needs the ability to get the writer schema from the broker. There is a workaround that we can send a REST request to query the schema, but it requires the admin permission.
### Modification
- Add the `Client::getSchemaInfoAsync` method to get the schema info asynchronously. When the schema of the given version does not exist, the `SchemaInfo` whose type is `NONE` will be returned.
- Add the `Message::getLongSchemaVersion` method to get the schema version of a message. The existing `getSchemaVersion` method is hard to use because it returns a byte array, which users need to know how to decode.
- Provide `fromBigEndianBytes` and `toBigEndianBytes` functions to perform conversion between a byte array and the long value of the schema version. Add `Int64SerDes` to test them.
- Fix the `LookupServiceTest` that initializes `client_` only once (even with the `GetParam()` method) so the HTTP URL was never tested.
- Add `testGetSchemaByVersion` to test `getSchemaInfoAsync`.
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc-required`
(Your PR needs to update docs and you will update later)
- [ ] `doc-not-needed`
(Please explain why)
- [x] `doc`
(Your PR contains doc changes)
- [ ] `doc-complete`
(Docs have been already added)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-cpp] shibd commented on a diff in pull request #257: Support get the SchemaInfo from a topic and the schema version
Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #257:
URL: https://github.com/apache/pulsar-client-cpp/pull/257#discussion_r1178685435
##########
include/pulsar/Message.h:
##########
@@ -177,7 +177,14 @@ class PULSAR_PUBLIC Message {
bool hasSchemaVersion() const;
/**
- * Get the schema version
+ * Get the schema version.
+ *
+ * @return the the schema version on success or -1 if the message does not have the schema version
+ */
+ int64_t getLongSchemaVersion() const;
Review Comment:
Get it, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-cpp] BewareMyPower commented on a diff in pull request #257: Support get the SchemaInfo from a topic and the schema version
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #257:
URL: https://github.com/apache/pulsar-client-cpp/pull/257#discussion_r1178544663
##########
include/pulsar/Message.h:
##########
@@ -177,7 +177,14 @@ class PULSAR_PUBLIC Message {
bool hasSchemaVersion() const;
/**
- * Get the schema version
+ * Get the schema version.
+ *
+ * @return the the schema version on success or -1 if the message does not have the schema version
+ */
+ int64_t getLongSchemaVersion() const;
Review Comment:
Yes. The Java API design is terrible. It really harms the user experience. Actually the byte array always represents a big-endian encoded 8 bytes integer, unless it's empty, which indicates no schema version. I believe it's a mistake made by the author. You can also see `Schemas#getSchemaInfo` in the `pulsar-admin` module, you can see the parameter type is long.
```java
SchemaInfo getSchemaInfo(String topic, long version) throws PulsarAdminException;
CompletableFuture<SchemaInfo> getSchemaInfoAsync(String topic, long version);
```
However, there is no document that tell users how to get the human-readable representation of the byte array. Returning a byte array is definitely a mistake. The Python client also wraps this API and returns a string to users. And the users can nearly do nothing with it.
I will open a discussion later in the mail list. But for now, I think it's good to add such a method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-cpp] BewareMyPower merged pull request #257: Support get the SchemaInfo from a topic and the schema version
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower merged PR #257:
URL: https://github.com/apache/pulsar-client-cpp/pull/257
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-cpp] BewareMyPower commented on a diff in pull request #257: Support get the SchemaInfo from a topic and the schema version
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #257:
URL: https://github.com/apache/pulsar-client-cpp/pull/257#discussion_r1178548743
##########
include/pulsar/Client.h:
##########
@@ -404,6 +404,19 @@ class PULSAR_PUBLIC Client {
*/
uint64_t getNumberOfConsumers();
+ /**
+ * Asynchronously get the SchemaInfo of a topic and a specific version.
+ *
+ * @topic the topic name
+ * @version the schema version byte array, see Message::getLongSchemaVersion.
+ * @callback the callback that is triggered when the SchemaInfo is retrieved successfully or not.
+ *
+ * NOTE: If there is no schema registered or the given version does not exist, a SchemaInfo whose type is
Review Comment:
In Java client, the `TopicNotFound` error is ignored and an empty optional of `SchemaInfo` is returned. So I thought it should not be treated as an exceptional case and users should process this error. I just use a `NONE` schema to replace an empty optional.
But I just thought again and IMO an error should be returned if we cannot get the schema for the given topic and version. So I will adopt the suggestion you made so that users should not handle the schema type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-cpp] shibd commented on a diff in pull request #257: Support get the SchemaInfo from a topic and the schema version
Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #257:
URL: https://github.com/apache/pulsar-client-cpp/pull/257#discussion_r1178056562
##########
include/pulsar/Client.h:
##########
@@ -404,6 +404,19 @@ class PULSAR_PUBLIC Client {
*/
uint64_t getNumberOfConsumers();
+ /**
+ * Asynchronously get the SchemaInfo of a topic and a specific version.
+ *
+ * @topic the topic name
+ * @version the schema version byte array, see Message::getLongSchemaVersion.
Review Comment:
This note does not match the parameter type.
##########
include/pulsar/Message.h:
##########
@@ -177,7 +177,14 @@ class PULSAR_PUBLIC Message {
bool hasSchemaVersion() const;
/**
- * Get the schema version
+ * Get the schema version.
+ *
+ * @return the the schema version on success or -1 if the message does not have the schema version
+ */
+ int64_t getLongSchemaVersion() const;
Review Comment:
For the Java client, it returns a bytes array. This may be a consensus that is already acceptable to users, is it necessary for us to change it to int?
##########
include/pulsar/Client.h:
##########
@@ -404,6 +404,19 @@ class PULSAR_PUBLIC Client {
*/
uint64_t getNumberOfConsumers();
+ /**
+ * Asynchronously get the SchemaInfo of a topic and a specific version.
+ *
+ * @topic the topic name
+ * @version the schema version byte array, see Message::getLongSchemaVersion.
+ * @callback the callback that is triggered when the SchemaInfo is retrieved successfully or not.
+ *
+ * NOTE: If there is no schema registered or the given version does not exist, a SchemaInfo whose type is
Review Comment:
I think we can add a new `Result` : `TopicSchemaNotFound`
When schema is not found, the Result on the callback is `TopicSchemaNotFound`.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-cpp] armanjupriya-er commented on pull request #257: Support get the SchemaInfo from a topic and the schema version
Posted by "armanjupriya-er (via GitHub)" <gi...@apache.org>.
armanjupriya-er commented on PR #257:
URL: https://github.com/apache/pulsar-client-cpp/pull/257#issuecomment-1700596974
> @armanjupriya-er The Pulsar CPP client version is no longer associated with the Pulsar version. So there is no need to "back merged". Just use the latest client
Thank you @BewareMyPower of the quick reply.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-cpp] RobertIndie commented on a diff in pull request #257: Support get the SchemaInfo from a topic and the schema version
Posted by "RobertIndie (via GitHub)" <gi...@apache.org>.
RobertIndie commented on code in PR #257:
URL: https://github.com/apache/pulsar-client-cpp/pull/257#discussion_r1177607224
##########
include/pulsar/Message.h:
##########
@@ -176,8 +176,16 @@ class PULSAR_PUBLIC Message {
*/
bool hasSchemaVersion() const;
+ /**
+ * Get the schema version.
+ *
+ * @return the the schema version on success or -1 if the message does not have the schema version
+ */
+ int64_t getLongSchemaVersion() const;
+
/**
* Get the schema version
+ * @deprecated Use getLongSchemaVersion instead
Review Comment:
It seems that we can also do it on the Java client. If we want to replace this public API, it's better to propose a PIP and change the Java client side first. I think we need to keep the API as consistent as possible with the Java client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-cpp] BewareMyPower commented on a diff in pull request #257: Support get the SchemaInfo from a topic and the schema version
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #257:
URL: https://github.com/apache/pulsar-client-cpp/pull/257#discussion_r1178552636
##########
include/pulsar/Client.h:
##########
@@ -404,6 +404,19 @@ class PULSAR_PUBLIC Client {
*/
uint64_t getNumberOfConsumers();
+ /**
+ * Asynchronously get the SchemaInfo of a topic and a specific version.
+ *
+ * @topic the topic name
+ * @version the schema version byte array, see Message::getLongSchemaVersion.
+ * @callback the callback that is triggered when the SchemaInfo is retrieved successfully or not.
+ *
+ * NOTE: If there is no schema registered or the given version does not exist, a SchemaInfo whose type is
Review Comment:
I won't add a new `Result`. The server side returns a `TopicNotFound` error. https://github.com/apache/pulsar/blob/f7c0b3c49c9ad8c28d0b00aa30d727850eb8bc04/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java#L999
So I use `ResultTopicNotFound` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-cpp] BewareMyPower commented on a diff in pull request #257: Support get the SchemaInfo from a topic and the schema version
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #257:
URL: https://github.com/apache/pulsar-client-cpp/pull/257#discussion_r1177651271
##########
include/pulsar/Message.h:
##########
@@ -176,8 +176,16 @@ class PULSAR_PUBLIC Message {
*/
bool hasSchemaVersion() const;
+ /**
+ * Get the schema version.
+ *
+ * @return the the schema version on success or -1 if the message does not have the schema version
+ */
+ int64_t getLongSchemaVersion() const;
+
/**
* Get the schema version
+ * @deprecated Use getLongSchemaVersion instead
Review Comment:
Okay, let's not mark it as deprecated in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-cpp] BewareMyPower commented on a diff in pull request #257: Support get the SchemaInfo from a topic and the schema version
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #257:
URL: https://github.com/apache/pulsar-client-cpp/pull/257#discussion_r1179067778
##########
include/pulsar/Message.h:
##########
@@ -176,8 +176,16 @@ class PULSAR_PUBLIC Message {
*/
bool hasSchemaVersion() const;
+ /**
+ * Get the schema version.
+ *
+ * @return the the schema version on success or -1 if the message does not have the schema version
+ */
+ int64_t getLongSchemaVersion() const;
+
/**
* Get the schema version
+ * @deprecated Use getLongSchemaVersion instead
Review Comment:
@RobertIndie PTAL again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-cpp] armanjupriya-er commented on pull request #257: Support get the SchemaInfo from a topic and the schema version
Posted by "armanjupriya-er (via GitHub)" <gi...@apache.org>.
armanjupriya-er commented on PR #257:
URL: https://github.com/apache/pulsar-client-cpp/pull/257#issuecomment-1698924826
@BewareMyPower : Can this PR be back merged to Pulsar 2.10 CPP client?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-cpp] BewareMyPower commented on pull request #257: Support get the SchemaInfo from a topic and the schema version
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #257:
URL: https://github.com/apache/pulsar-client-cpp/pull/257#issuecomment-1700249682
@armanjupriya-er The Pulsar CPP client version is no longer associated with the Pulsar version. So there is no need to "back merged". Just use the latest client
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org