You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/12/30 06:18:32 UTC
[GitHub] [pulsar] kyrozetera opened a new issue #9089: Pulsar not respecting isSchemaValidationEnforced=false on startup
kyrozetera opened a new issue #9089:
URL: https://github.com/apache/pulsar/issues/9089
**Describe the bug**
Pulsar is not respecting the `isSchemaValidationEnforced=false` or `isAllowAutoUpdateSchema=true` in the broker.conf on startup. Though validation appears to be disabled when checking the value through the API, it still does the validation and fails on conflicting schemas if the schema has been updated. When manually disabling the validation through the API, then it begins to function as expected. To get `isAllowAutoUpdateSchema` to function properly, it was a bit more involved as I had to disable and enable both it and the schema validation through the API before it started working.
I'm using the debezium connector with postgres, for pulsar 2.7.0 and this is where I'm seeing this behavior (same behavior in 2.6.1 & 2.6.2).
**To Reproduce**
Steps to reproduce the behavior:
1. Setup Debezium source connector following by https://pulsar.apache.org/docs/en/io-cdc-debezium/
2. Ensure config is set `isSchemaValidationEnforced=false` in broker.conf or standalone.conf
3. Start pulsar standalone `bin/pulsar standalone`
4. Start debezium connector `bin/pulsar-admin source localrun --source-config-file conf/debezium-postgres-source-config.yaml`
5. Check schema validation: `bin/pulsar-admin namespaces get-schema-validation-enforce public/default` - `false`
6. Create table and insert record:
```sql
CREATE TABLE public.test_table (
id int4 NOT NULL GENERATED BY DEFAULT AS IDENTITY,
campaign_id int4 NOT NULL,
"name" varchar NULL,
start_date timestamp NOT NULL,
end_date timestamp NOT NULL,
CONSTRAINT test_table_campaign_id_name_unique UNIQUE (campaign_id, name),
CONSTRAINT pk_test_table_id PRIMARY KEY (id)
);
INSERT INTO test_table (campaign_id, name, start_date, end_date) VALUES (1, 'test', '2020-12-01', '2020-12-31');
```
7. Alter column, then insert again:
```sql
ALTER TABLE test_table ALTER COLUMN end_date DROP NOT NULL;
INSERT INTO test_table (campaign_id, name, start_date, end_date) VALUES (2, 'test', '2020-12-01', '2020-12-31');
```
8. See error:
```
00:06:02.592 [pulsar-client-io-1-2] WARN org.apache.pulsar.client.impl.ProducerImpl - [public/default/koddi-console-1.public.test_table] [standalone-0-2] GetOrCreateSchema error
org.apache.pulsar.client.api.PulsarClientException$IncompatibleSchemaException: org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException: org.apache.avro.SchemaValidationException: Unable to read schema:
```
9. Disable via API: `bin/pulsar-admin namespaces set-schema-validation-enforce public/default --disable`
10. Restart debezium connector
11. Insert again and see no validation error in debezium/pulsar
```sql
insert into test_table (campaign_id, name, start_date, end_date) VALUES (3, 'test', '2020-12-01', '2020-12-31');
```
**Expected behavior**
The schema should not be validated, or it should auto update when enabled
**Screenshots**
N/A
**Desktop (please complete the following information):**
- OS: Linux (Ubuntu 20.04 / Fedora / Debian10)
**Additional context**
Having the same issue with a standalone instance locally and a pulsar cluster
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9089: Pulsar not respecting isSchemaValidationEnforced=false on startup
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9089:
URL: https://github.com/apache/pulsar/issues/9089#issuecomment-1058894360
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9089: Pulsar not respecting isSchemaValidationEnforced=false on startup
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9089:
URL: https://github.com/apache/pulsar/issues/9089#issuecomment-770235992
@kyontan Sorry for the late response and thanks for the feedback. We will take a look soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9089: Pulsar not respecting isSchemaValidationEnforced=false on startup
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9089:
URL: https://github.com/apache/pulsar/issues/9089#issuecomment-788547011
@kyontan The `isSchemaValidationEnforced` is the control the schema validation which the producer without schema but the topic has schema. This means if the topic with schema but the produce does not have a schema and `isSchemaValidationEnforced=true`, the broker will reject the producer. Otherwise, the broker will do the schema compatibility check, from your description, I think the problem is you have dropped a `NOT NULL` COLUMN, this will break the forward compatibility, since the default schema compatibility policy is FULL(forward and backward), so you got the `IncompatibleSchemaException`, you can try to update the schema compatibility policy to BACKWARD.
For more details you can check http://pulsar.apache.org/docs/en/schema-evolution-compatibility/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org