You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/05/06 08:13:20 UTC
[GitHub] [pulsar] bradfordChiang opened a new issue #6889: Provide an injection point to use personal ObjectMapper in JsonSchema
bradfordChiang opened a new issue #6889:
URL: https://github.com/apache/pulsar/issues/6889
Currently the ObjectMapper in org.apache.pulsar.client.impl.schema.JSONSchema is private and hard coded, which prevents us from using lots of Java classes like JavaTime, Guava etc.
It would be great if you can provide a set method for us to change the ObjectMapper reference.
And ObjectMapper is a thread-safe class, is the ThreadLocal necessary?
`
static ObjectMapper mapper = new ObjectMapper();
static {
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
mapper.setSerializationInclusion(JsonInclude.Include.NON_NULL);
}
public static void setObjectMapper(ObjectMapper objectMapper) {
Objects.requireNonNull(objectMapper, "'objectMapper' must not be null");
mapper = objectMapper;
}
public SchemaInfo getBackwardsCompatibleJsonSchemaInfo() {
SchemaInfo backwardsCompatibleSchemaInfo;
try {
JsonSchemaGenerator schemaGen = new JsonSchemaGenerator(mapper);
JsonSchema jsonBackwardsCompatibleSchema = schemaGen.generateSchema(pojo);
backwardsCompatibleSchemaInfo = new SchemaInfo();
backwardsCompatibleSchemaInfo.setName("");
backwardsCompatibleSchemaInfo.setProperties(schemaInfo.getProperties());
backwardsCompatibleSchemaInfo.setType(SchemaType.JSON);
backwardsCompatibleSchemaInfo.setSchema(mapper.writeValueAsBytes(jsonBackwardsCompatibleSchema));
} catch (JsonProcessingException ex) {
throw new RuntimeException(ex);
}
return backwardsCompatibleSchemaInfo;
}
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jdimeo commented on issue #6889: Provide an injection point to use personal ObjectMapper in JsonSchema
Posted by GitBox <gi...@apache.org>.
jdimeo commented on issue #6889:
URL: https://github.com/apache/pulsar/issues/6889#issuecomment-887997379
@lhotari Something is still fishy, and I can't quite get to the bottom of it. I injected my own `ObjectMapper` using the `SchemaDefinitionBuilder` as recommended and I even confirmed it was being used with ugly logging in the function itself (using the new output message builder to send a message):
```
23:27:08.576 [public/default/fn] INFO org.ci.cdm.fn.util.JacksonJsonWriter - WRITING MESSAGE WITH CORRECT OBJECT MAPPER {"id":"2052194","timestamp":"2021-03-31T19:30:17.344+00:00","action":"Update", ...
```
So the correct bytes are being generated (notably timestamps as UTC and using `@JsonValue` for the enum `action`).
*However*, no matter *what* I do, I always get these bytes in the client:
```
Cannot construct instance of `org.ci.cdm.event.ChangeEvent$Action`, problem: UPDATE
at [Source: (byte[])"{"id":"2052194","timestamp":1617219017344,"action":"UPDATE",
```
- I'm using the same schema in both my client test code and the function code
- Even with a vanilla `ObjectMapper`, my enum is annotated with a `@JsonValue` for the title case "Update" - why is it being ignored?
- Is something like gson or org.json getting in the way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MaitreyaBuddha commented on issue #6889: Provide an injection point to use personal ObjectMapper in JsonSchema
Posted by GitBox <gi...@apache.org>.
MaitreyaBuddha commented on issue #6889:
URL: https://github.com/apache/pulsar/issues/6889#issuecomment-705147125
I am hitting a similar issue because that configured ObjectMapper does not support deserializing Instant. I'm trying to add the necessary `mapper.configure` line to fix it because I want to utilize Pulsar's deserialization instead of making my own mapper and deserializing. An easy workaround is of course to not use JsonSchema and do deserialize yourself with your ObjectMapper. I'm curious why you would want to make your own mapper object but not use it yourself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jdimeo edited a comment on issue #6889: Provide an injection point to use personal ObjectMapper in JsonSchema
Posted by GitBox <gi...@apache.org>.
jdimeo edited a comment on issue #6889:
URL: https://github.com/apache/pulsar/issues/6889#issuecomment-887997379
@lhotari Something is still fishy, and I can't quite get to the bottom of it. I injected my own `ObjectMapper` using the `SchemaDefinitionBuilder` as recommended and I even confirmed it was being used with ugly logging in the function itself (using the new output message builder to send a message):
```
23:27:08.576 [public/default/fn] INFO org...JacksonJsonWriter - WRITING MESSAGE WITH CORRECT OBJECT MAPPER {"id":"2052194","timestamp":"2021-03-31T19:30:17.344+00:00","action":"Update", ...
```
So the correct bytes are being generated (notably timestamps as UTC and using `@JsonValue` for the enum `action`).
*However*, no matter *what* I do, I always get these bytes in the client:
```
Cannot construct instance of `org...ChangeEvent$Action`, problem: UPDATE
at [Source: (byte[])"{"id":"2052194","timestamp":1617219017344,"action":"UPDATE",
```
- I'm using the same schema in both my client test code and the function code
- Even with a vanilla `ObjectMapper`, my enum is annotated with a `@JsonValue` for the title case "Update" - why is it being ignored?
- Is something like gson or org.json getting in the way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Persi commented on issue #6889: Provide an injection point to use personal ObjectMapper in JsonSchema
Posted by GitBox <gi...@apache.org>.
Persi commented on issue #6889:
URL: https://github.com/apache/pulsar/issues/6889#issuecomment-625041039
This is a copy of #6842 which I just renamed to make it clearer and easier to find ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jdimeo edited a comment on issue #6889: Provide an injection point to use personal ObjectMapper in JsonSchema
Posted by GitBox <gi...@apache.org>.
jdimeo edited a comment on issue #6889:
URL: https://github.com/apache/pulsar/issues/6889#issuecomment-887997379
@lhotari Something is still fishy, and I can't quite get to the bottom of it. I injected my own `ObjectMapper` using the `SchemaDefinitionBuilder` as recommended and I even confirmed it was being used with ugly logging in the function itself (using the new output message builder to send a message):
```
23:27:08.576 [public/default/fn] INFO org...JacksonJsonWriter - WRITING MESSAGE WITH CORRECT OBJECT MAPPER {"id":"2052194","timestamp":"2021-03-31T19:30:17.344+00:00","action":"Update", ...
```
So the correct bytes are being generated (notably timestamps as UTC and using `@JsonValue` for the enum `action`).
*However*, no matter *what* I do, I always get these bytes in the client:
```
Cannot construct instance of `org...ChangeEvent$Action`, problem: UPDATE
at [Source: (byte[])"{"id":"2052194","timestamp":1617219017344,"action":"UPDATE",
```
Notably, the timestamps are back to ms since epoch and the enum is using the `name()`
- I'm using the same schema in both my client test code and the function code
- Even with a vanilla `ObjectMapper`, my enum is annotated with a `@JsonValue` for the title case "Update" - why is it being ignored?
- Is something like gson or org.json getting in the way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #6889: Provide an injection point to use personal ObjectMapper in JsonSchema
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #6889:
URL: https://github.com/apache/pulsar/issues/6889#issuecomment-713647291
This feature has already been implemented by #6842 which is included since 2.6.0+.
There's a Kotlin code example in https://github.com/apache/pulsar/issues/6528#issuecomment-701410483 about how to use it. I haven't tried it myself, but there's a note that it doesn't work with the shaded client and you should use pulsar-client-original dependency instead of pulsar-client. I guess #6902 is to cover the shading issues. As a workaround you should be able to use the unshaded pulsar-client-original dependency.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jdimeo edited a comment on issue #6889: Provide an injection point to use personal ObjectMapper in JsonSchema
Posted by GitBox <gi...@apache.org>.
jdimeo edited a comment on issue #6889:
URL: https://github.com/apache/pulsar/issues/6889#issuecomment-887997379
**EDIT:** Apologies. In my testing, I was using the same topic and so the schema was "locked in" to the previous schema before I introduced the custom `ObjectMapper`. I can confirm that this method does work, though is a bit verbose and explodes your function .jar size since you need the client classes in there to leverage a custom schema (including stuff like Netty for byte buffers and stuff, strangely).
Original:
@lhotari Something is still fishy, and I can't quite get to the bottom of it. I injected my own `ObjectMapper` using the `SchemaDefinitionBuilder` as recommended and I even confirmed it was being used with ugly logging in the function itself (using the new output message builder to send a message):
```
23:27:08.576 [public/default/fn] INFO org...JacksonJsonWriter - WRITING MESSAGE WITH CORRECT OBJECT MAPPER {"id":"2052194","timestamp":"2021-03-31T19:30:17.344+00:00","action":"Update", ...
```
So the correct bytes are being generated (notably timestamps as UTC and using `@JsonValue` for the enum `action`).
*However*, no matter *what* I do, I always get these bytes in the client:
```
Cannot construct instance of `org...ChangeEvent$Action`, problem: UPDATE
at [Source: (byte[])"{"id":"2052194","timestamp":1617219017344,"action":"UPDATE",
```
Notably, the timestamps are back to ms since epoch and the enum is using the `name()`
- I'm using the same schema in both my client test code and the function code
- Even with a vanilla `ObjectMapper`, my enum is annotated with a `@JsonValue` for the title case "Update" - why is it being ignored?
- Is something like gson or org.json getting in the way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org