You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "NurramoX (via GitHub)" <gi...@apache.org> on 2024/11/06 17:24:20 UTC

[D] Setting Input Topic Schemas for Pulsar Sink [pulsar]

GitHub user NurramoX created a discussion: Setting Input Topic Schemas for Pulsar Sink

I'm trying to configure schemas for input topics in a Pulsar sink connector, similar to how we can set the schema for output topics in function using `outputSchemaType`.

I've explored two approaches:

- Using `sinkConfig.setTopicToSchemaType`:
  - Attempted to set ConsumerConfig
  - Tried setting the same schema type as used in outputSchemaType via the schemaType field of ConsumerConfig


- Using `sinkConfig.setTopicToSchemaType`:
  - almost the same procedure as for InputSpecs

Both approaches result in the following schema type mismatch error:
`Schema type mismatch org.apache.pulsar.client.api.schema.GenericObject vs com.example.dtos.SimpleObject`

### Questions

1. Is this expected behavior? Can we only set custom schemas (classes implementing the schema interface) on output topics?
2. If it is possible to set input topic schemas, what's the correct approach?


### Additional Context

This is the Schema that I am using (kept it simple for demo purposes):

```java
public class SimpleSchema implements Schema<SimpleObject> {
    private final Schema<SimpleObject> avroSchema = Schema.AVRO(SimpleObject.class);
    
    @Override
    public byte[] encode(SimpleObject message) { return avroSchema.encode(message); }
    
    @Override
    public byte[] decode(SimpleObject message) { return avroSchema.decode(message); }
    
    @Override
    public SchemaInfo getSchemaInfo() { return avroSchema.getSchemaInfo(); }
    
    @Override
    public Schema<SimpleObject> clone() { return new SimpleSchema(); }
}
```

GitHub link: https://github.com/apache/pulsar/discussions/23573

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org

Re: [D] Setting Input Topic Schemas for Pulsar Sink [pulsar]

Posted by "NurramoX (via GitHub)" <gi...@apache.org>.
GitHub user NurramoX edited a discussion: Setting Input Topic Schemas for Pulsar Sink

I'm trying to configure schemas for input topics in a Pulsar sink connector, similar to how we can set the schema for output topics in function using `outputSchemaType`.

I've explored two approaches:

- Using `sinkConfig.setInputSpecs`:
  - Attempted to set ConsumerConfig
  - Tried setting the same schema type as used in outputSchemaType via the schemaType field of ConsumerConfig


- Using `sinkConfig.setTopicToSchemaType`:
  - almost the same procedure as for InputSpecs

Both approaches result in the following schema type mismatch error:
`Schema type mismatch org.apache.pulsar.client.api.schema.GenericObject vs com.example.dtos.SimpleObject`

### Questions

1. Is this expected behavior? Can we only set custom schemas (classes implementing the schema interface) on output topics?
2. If it is possible to set input topic schemas, what's the correct approach?


### Additional Context

This is the Schema that I am using (kept it simple for demo purposes):

```java
public class SimpleSchema implements Schema<SimpleObject> {
    private final Schema<SimpleObject> avroSchema = Schema.AVRO(SimpleObject.class);
    
    @Override
    public byte[] encode(SimpleObject message) { return avroSchema.encode(message); }
    
    @Override
    public byte[] decode(SimpleObject message) { return avroSchema.decode(message); }
    
    @Override
    public SchemaInfo getSchemaInfo() { return avroSchema.getSchemaInfo(); }
    
    @Override
    public Schema<SimpleObject> clone() { return new SimpleSchema(); }
}
```

GitHub link: https://github.com/apache/pulsar/discussions/23573

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org