You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "smbecker (via GitHub)" <gi...@apache.org> on 2025/03/11 20:47:08 UTC

[I] Breaking change with using multiple schemas on a topic [pulsar-dotpulsar]

smbecker opened a new issue, #260:
URL: https://github.com/apache/pulsar-dotpulsar/issues/260

   ### Description
   
   I have a topic that was created using `NewConsumer(Schema.String)`. However, for retries and dead letters, I frequently create the producers using `NewProducer()` since I don't want to pay the `Encode`/`Decode` penalty since it is just forwarding the message data from the consumer into the retry queue. After upgrading to 4.2, I noticed that my retry producers were faulting with a `IncompatibleSchemaException`.
   
   ### Reproduction Steps
   
   ```c#
   [Fact]
   public async Task can_publish_retry_to_consumer() {
   	var pulsarContainer = new PulsarBuilder().Build();
   	await pulsarContainer.StartAsync();
   	try {
   		await using var client = PulsarClient.Builder()
   			.ServiceUrl(new Uri(pulsarContainer.GetBrokerAddress()))
   			.Build();
   		await using var originProducer = client.NewProducer(Schema.String)
   			.Topic("persistent://public/default/test")
   			.Create();
   		await using var originConsumer = client.NewConsumer(Schema.String)
   			.Topic("persistent://public/default/test")
   			.SubscriptionName("test")
   			.SubscriptionType(SubscriptionType.Exclusive)
   			.Create();
   		await using var retryProducer = client.NewProducer()
   			.Topic("persistent://public/default/test")
   			.Create();
   
   		await originProducer.Send("test");
   		var testMessage = await originConsumer.Receive();
   		Assert.Equal("test", testMessage.Value());
   
   		await retryProducer.Send(testMessage.Data);
   
   		testMessage = await originConsumer.Receive();
   		Assert.Equal("test", testMessage.Value());
   	} finally {
   		await pulsarContainer.StopAsync();
   	}
   }
   ```
   
   ### Expected behavior
   
   Technically, this is more accurate in creating the topic with the correct schema. However, it is a change in behavior and not one that was expected. Perhaps a flag that allows you to opt-out of setting the schema at the server-side topic level to preserve the old behavior.
   
   ### Actual behavior
   
   Throws a `ProducerFaultedException` with the following as the `InnerException`:
   
   ```
   DotPulsar.Exceptions.IncompatibleSchemaException: org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException: Incompatible schema: exists schema type STRING, new schema type NONE caused by org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException: Incompatible schema: exists schema type STRING, new schema type NONE
   ```
   
   ### Regression?
   
   I would consider this a regression as it is a change in behavior and not one that was expected.
   
   ### Known Workarounds
   
   One workaround is to create all producers and consumers using `ReadOnlySequence<byte>` and handle encoding and decoding yourself. Another workaround is to create a new `ISchema<string>` with `Schema.ByteSequence.SchemaInfo`.
   
   ```c#
   private sealed class CustomStringSchema : ISchema<string>
   {
   	public static ISchema<string> Instance { get; } = new CustomStringSchema();
   
   	public SchemaInfo SchemaInfo => Schema.ByteSequence.SchemaInfo;
   
   	public string Decode(ReadOnlySequence<byte> bytes, byte[]? schemaVersion = null) {
   		return Schema.String.Decode(bytes, schemaVersion);
   	}
   
   	public ReadOnlySequence<byte> Encode(string message) {
   		return Schema.String.Encode(message);
   	}
   }
   ```
   
   ### Configuration
   
   * Which version of .NET is the code running on? `net9.0`
   * Which version of DotPulsar are you using? `4.2.0`
   * What OS and version, and what distro if applicable? `Ubuntu 24.04`
   * What is the architecture (x64, x86, ARM, ARM64)? `x64`
   * Do you know whether it is specific to that configuration? It doesn't appear to be related to OS/.NET configuration at all. It seems to be a breaking change from [this PR](https://github.com/apache/pulsar-dotpulsar/pull/253)
   
   ### Other information
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Breaking change with using multiple schemas on a topic [pulsar-dotpulsar]

Posted by "smbecker (via GitHub)" <gi...@apache.org>.
smbecker commented on issue #260:
URL: https://github.com/apache/pulsar-dotpulsar/issues/260#issuecomment-2717801189

   I confirmed that 4.2.1-rc.1 fixed this issue. Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Breaking change with using multiple schemas on a topic [pulsar-dotpulsar]

Posted by "smbecker (via GitHub)" <gi...@apache.org>.
smbecker closed issue #260: Breaking change with using multiple schemas on a topic
URL: https://github.com/apache/pulsar-dotpulsar/issues/260


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Breaking change with using multiple schemas on a topic [pulsar-dotpulsar]

Posted by "blankensteiner (via GitHub)" <gi...@apache.org>.
blankensteiner commented on issue #260:
URL: https://github.com/apache/pulsar-dotpulsar/issues/260#issuecomment-2717623222

   Hi @blankensteiner from the past. I second that idea... great minds think alike (ignoring the "though fools seldom differ" part). I have committed a fix and will release 4.2.1-rc.1.
   If you (@smbecker) have a better idea, we can just create a second RC :-)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Breaking change with using multiple schemas on a topic [pulsar-dotpulsar]

Posted by "blankensteiner (via GitHub)" <gi...@apache.org>.
blankensteiner commented on issue #260:
URL: https://github.com/apache/pulsar-dotpulsar/issues/260#issuecomment-2717331218

   Hi @smbecker 
   Thanks for reporting this.
   I suggest we fix this by NOT sending the schema when the Schema.Type is 'None', which it is for ByteArraySchema and ByteSequenceSchema.
   Does that sound like a good solution?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org