You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/08/03 10:21:39 UTC
[GitHub] [pulsar] BewareMyPower opened a new issue #7728: How to create a topic without schema or delete the schema?
BewareMyPower opened a new issue #7728:
URL: https://github.com/apache/pulsar/issues/7728
I have encountered schema problems recently. eg, creating a consumer with `Schema.STRING`:
```java
try (PulsarClient client = PulsarClient.builder() //
.serviceUrl("pulsar://localhost:6650") //
.build()) {
try (Consumer<String> consumer = client.newConsumer(Schema.STRING) //
.topic("Foo") //
.subscriptionName("test-sub") //
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest) //
.subscribe()) {
log.info("Subscribe successfully");
}
} catch (PulsarClientException e) {
log.error("{}", e.getMessage(), e);
}
```
The log is:
> org.apache.pulsar.client.api.PulsarClientException$IncompatibleSchemaException: Topic does not have schema to check
Finally I found some topics may not have associated schema, see `SchemaRegistryServiceImpl` in `broker.service.schema` package:
```java
public CompletableFuture<Void> checkConsumerCompatibility(String schemaId, SchemaData schemaData,
SchemaCompatibilityStrategy strategy) {
return getSchema(schemaId).thenCompose(existingSchema -> {
if (existingSchema != null && !existingSchema.schema.isDeleted()) {
// ...
} else { // existingSchema is null or it's deleted
return FutureUtil.failedFuture(new IncompatibleSchemaException("Topic does not have schema to check"));
}
});
}
```
I thought it's because those topics have been produced by C++ client without schema. But it seems that I'm wrong. I created a new topic and used C++ client to produce some messages, but a consumer with string schema subscribed successfully. I've also trying producing some messages with Java producer without schema, but the result were the same.
In my private cluster for test, there're still 2 topics that couldn't be subscribed with string schema. But I didn't know how did I make it happen.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7728: Automatically created topics have no valid schema info
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-696498204
Discussed with @BewareMyPower, we might need to improve the documentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet edited a comment on issue #7728: Automatically created topics have no valid schema info
Posted by GitBox <gi...@apache.org>.
Anonymitaet edited a comment on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-763453471
@Huanli-Meng thanks for your reply, then I remove the `doc` related tags.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #7728: How to create a topic without schema or delete the schema?
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-668049346
@codelipenghui It's currently 2.6.0, but there's an update from 2.5.1 to 2.6.0 a few days ago.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower edited a comment on issue #7728: Automatically created partitioned topics have no valid schema info
Posted by GitBox <gi...@apache.org>.
BewareMyPower edited a comment on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-689286006
@codelipenghui @sijie I reproduced the issue accidentally. If the broker's `allowAutoTopicCreationType` is `partitioned`, then the automatically created topic has no schema.
It's easy to reproduce:
1. Run a consumer to subscribe a topic that doesn't exist now, e.g.
```java
try (Consumer<byte[]> consumer = client.newConsumer() //
.topic("PulsarBar") // PulsarBar doesn't exist
.subscriptionName("my-sub") //
.subscribe()) {
```
2. Check the topic's schema
```bash
$ ./bin/pulsar-admin schemas get "persistent://public/default/PulsarBar"
HTTP 404 Not Found
Reason: HTTP 404 Not Found
$ ./bin/pulsar-admin schemas get "persistent://public/default/PulsarBar-partition-0"
HTTP 404 Not Found
Reason: HTTP 404 Not Found
```
The broker logs:
```
11:36:15.409 [ForkJoinPool.commonPool-worker-25] INFO org.eclipse.jetty.server.RequestLog - 10.208.65.74 - - [09/Sep/2020:11:36:15 +0800] "GET /admin/v2/schemas/public/default/PulsarBar/schema?authoritative=true HTTP/1.1" 404 0 "-" "Pulsar-Java-v2.6.0" 4
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #7728: How to create a topic without schema or delete the schema?
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-668346967
@sijie The results are both `HTTP 404 Not Found`. The broker's log is also 404
```
10:39:40.284 [ForkJoinPool.commonPool-worker-14] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [04/Aug/2020:10:39:40 +0800] "GET /admin/v2/schemas/public/default/ParBar/schema HTTP/1.1" 404 0 "-" "Pulsar-Java-v2.6.0" 4
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Huanli-Meng commented on issue #7728: Automatically created topics have no valid schema info
Posted by GitBox <gi...@apache.org>.
Huanli-Meng commented on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-763451599
> @BewareMyPower need to update docs?
confirmed with @BewareMyPower, it is a code issue. no doc needs to be updated,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower edited a comment on issue #7728: How to create a topic without schema or delete the schema?
Posted by GitBox <gi...@apache.org>.
BewareMyPower edited a comment on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-668346967
@sijie The results are both `HTTP 404 Not Found`. The broker's log is also 404
> 10:39:40.284 [ForkJoinPool.commonPool-worker-14] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [04/Aug/2020:10:39:40 +0800] "GET /admin/v2/schemas/public/default/ParBar/schema HTTP/1.1" 404 0 "-" "Pulsar-Java-v2.6.0" 4
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on issue #7728: Automatically created topics have no valid schema info
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-763453471
@Huanli-Meng thanks for your reply, then I remove the `component/documentation` tag
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #7728: How to create a topic without schema or delete the schema?
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-668043495
@BewareMyPower Which version of the broker are you using?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Huanli-Meng commented on issue #7728: Automatically created topics have no valid schema info
Posted by GitBox <gi...@apache.org>.
Huanli-Meng commented on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-699860129
Confirmed with @BewareMyPower , he thoughts that it might be a code issue instead of a document issue. He will update the code and will ping @Huanli-Meng if the doc requires update.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] TIBCOeddie commented on issue #7728: Automatically created topics have no valid schema info
Posted by GitBox <gi...@apache.org>.
TIBCOeddie commented on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-790743886
I've hit this as well. A consumer shouldn't so easily leave a server-side resource ( a topic ) in an undesirable state like this...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #7728: Automatically created topics have no valid schema info
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-689563807
The description I've said before is not accurency. If I send a PUT request to create a topic, querying the topic's schema will also returns 404. But a consumer with string schema could still consume it. After that, the topic's schema became string schema.
However, if a topic was created automatically by a consumer without schema, then a consumer with string schema would fail with `Topic does not have schema to check`.
I'm a little confused with
> If these 2 topics don't have any schemas, it is expected to fail with Incompatible schema if you try to subscribe to them using Schema.STRING.
I've read some source code, if the broker only handled admin's create topic request, it would just call `BrokerService.getTopic` and wait until the returned future completed. But if the broker handled client's subscribe command while the topic didn't exist, the broker would do some more, which might lead to the difference behavior.
It looks like a topic without schema has two different states. I'm not familiar with schema, could you give me some help? @sijie
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #7728: Automatically created partitioned topics have no valid schema info
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-689286006
@codelipenghui @sijie I reproduced the issue accidentally. If the broker's `allowAutoTopicCreationType` is `partitioned`, then the automatically created topic has no schema.
It's easy to reproduce:
1. Run a consumer to subscribe a topic that doesn't exist now, e.g.:
```java
try (Consumer<byte[]> consumer = client.newConsumer() //
.topic("PulsarBar") // PulsarBar doesn't exist
.subscriptionName("my-sub") //
.subscribe()) {
```
2. Check the topic's schema
```bash
$ ./bin/pulsar-admin schemas get "persistent://public/default/PulsarBar"
HTTP 404 Not Found
Reason: HTTP 404 Not Found
$ ./bin/pulsar-admin schemas get "persistent://public/default/PulsarBar-partition-0"
HTTP 404 Not Found
Reason: HTTP 404 Not Found
```
The broker logs:
```
11:36:15.409 [ForkJoinPool.commonPool-worker-25] INFO org.eclipse.jetty.server.RequestLog - 10.208.65.74 - - [09/Sep/2020:11:36:15 +0800] "GET /admin/v2/schemas/public/default/PulsarBar/schema?authoritative=true HTTP/1.1" 404 0 "-" "Pulsar-Java-v2.6.0" 4
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7728: Automatically created partitioned topics have no valid schema info
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-689318148
@BewareMyPower If you have identified the problem, are you interested in contributing a fix?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7728: Automatically created topics have no valid schema info
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-696498204
Discussed with @BewareMyPower, we might need to improve the documentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7728: How to create a topic without schema or delete the schema?
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-668330567
> In my private cluster for test, there're still 2 topics that couldn't be subscribed with string schema. But I didn't know how did I make it happen.
Can you run `pulsar-admin schemas get` to get schemas of these 2 topics? If these 2 topics don't have any schemas, it is expected to fail with Incompatible schema if you try to subscribe to them using Schema.STRING.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on issue #7728: Automatically created topics have no valid schema info
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-763449056
@BewareMyPower need to update docs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #7728: Automatically created partitioned topics have no valid schema info
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-689324801
@sijie Yes, I'll take some time to look into the problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #7728: Automatically created topics have no valid schema info
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #7728:
URL: https://github.com/apache/pulsar/issues/7728#issuecomment-699799122
It looks like there's some wrong when broker handles subscribe request. Here's my test code:
```java
final static String serviceUrl = "pulsar://localhost:6650";
final static String webServiceUrl = "http://localhost:8080";
static void checkSchema(String topic) {
try (PulsarAdmin admin = PulsarAdmin.builder().serviceHttpUrl(webServiceUrl).build()) {
System.out.println("schema of " + topic + ": " + admin.schemas().getSchemaInfo(topic));
} catch (PulsarClientException | PulsarAdminException e) {
System.err.println("getSchemaInfo failed for " + topic + ": " + e.getMessage());
}
}
static void producerNoSchemaToCheck(String topic) {
try (PulsarClient client = PulsarClient.builder().serviceUrl(serviceUrl).build()) {
// 1. byte[] (without schema)
try (Producer<byte[]> producer = client.newProducer(Schema.BYTES).topic(topic).create()) {
System.out.println("Producer<byte[]>: " + producer.getProducerName());
}
checkSchema(topic);
// 2. String (with schema) => OK
try (Producer<String> producer = client.newProducer(Schema.STRING).topic(topic).create()) {
System.out.println("Producer<String>: " + producer.getProducerName());
}
checkSchema(topic);
} catch (PulsarClientException e) {
e.printStackTrace();
}
}
static void consumerNoSchemaToCheck(String topic) {
try (PulsarClient client = PulsarClient.builder().serviceUrl(serviceUrl).build()) {
// 1. byte[] (without schema)
try (Consumer<byte[]> consumer = client.newConsumer(Schema.BYTES).topic(topic).subscriptionName("my-sub").subscribe()) {
System.out.println("Consumer<byte[]>: " + consumer.getConsumerName());
}
checkSchema(topic);
// 2. String (with schema) => Topic does not have schema to check
try (Consumer<String> consumer = client.newConsumer(Schema.STRING).topic(topic).subscriptionName("my-sub").subscribe()) {
System.out.println("Consumer<String>: " + consumer.getConsumerName());
}
checkSchema(topic);
} catch (PulsarClientException e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
long t = System.currentTimeMillis();
consumerNoSchemaToCheck("my-topic-" + t);
producerNoSchemaToCheck("my-topic-" + (t + 1));
}
```
The output:
```
Consumer<byte[]>: b1669
getSchemaInfo failed for my-topic-1601273689739: HTTP 404 Not Found
org.apache.pulsar.client.api.PulsarClientException$IncompatibleSchemaException: Topic does not have schema to check
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:862)
at org.apache.pulsar.client.impl.ConsumerBuilderImpl.subscribe(ConsumerBuilderImpl.java:101)
at SchemaDemo.consumerNoSchemaToCheck(SchemaDemo.java:46)
at SchemaDemo.main(SchemaDemo.java:57)
Producer<byte[]>: standalone-0-24
getSchemaInfo failed for my-topic-1601273689740: HTTP 404 Not Found
Producer<String>: standalone-0-25
schema of my-topic-1601273689740: {
"name": "my-topic-1601273689740",
"schema": "",
"type": "STRING",
"properties": {}
}
```
- consumer:
1. Create a `Consumer<byte[]>` to create a new topic and subscribe it;
2. Check the schema: 404;
3. Create a `Consumer<String>` to subscribe the same topic, **failed: Topic does not have schema to check**.
- producer:
1. Create a `Producer<byte[]>` to create a new topic;
2. Check the schema: 404;
3. Create a `Producer<String>` to specify the same topic;
4. Check the schema: STRING.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #7728: Automatically created topics have no valid schema info
Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #7728:
URL: https://github.com/apache/pulsar/issues/7728
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org