You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Michael A. Smith (Jira)" <ji...@apache.org> on 2019/11/03 02:23:00 UTC
[jira] [Comment Edited] (AVRO-2411) RecordSchema with logicalType
field is not JSON serializable
[ https://issues.apache.org/jira/browse/AVRO-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16965541#comment-16965541 ]
Michael A. Smith edited comment on AVRO-2411 at 11/3/19 2:22 AM:
-----------------------------------------------------------------
I see. This was caused by the changes in AVRO-2234. Sorry for the inconvenience.
If you call {{json.dumps}} directly on a schema or on the output of {{Schema.to_json()}}, you'll see errors like you describe. But you probably should not call {{json.dumps}} directly on a schema instance. The {{__str__}} implementation of a schema object is exactly {{json.dumps}}, except that it includes an implementation of {{MappingProxyEncoder}} to solve this problem.
{code:python}
>>> import avro.schema, json, pprint
>>> example = avro.schema.parse('{"type":"record","name":"example","fields":[{"name":"exfield","type":{"type":"long","logicalType":"timestamp-millis"}}]}')
>>> schema_dict = example.to_json()
>>> pprint.pprint(schema_dict)
{'fields': [{'name': 'exfield',
'type': mappingproxy({'logicalType': 'timestamp-millis',
'type': 'long'})}],
'name': 'example',
'type': 'record'}
>>> json.dumps(example)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.5/json/__init__.py", line 230, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python3.5/json/encoder.py", line 198, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python3.5/json/encoder.py", line 256, in iterencode
return _iterencode(o, 0)
File "/usr/lib/python3.5/json/encoder.py", line 179, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <avro.schema.RecordSchema object at 0x7fcecdd53518> is not JSON serializable
{code}
...but
{code:python}
>>> print(str(example))
{"fields": [{"name": "exfield", "type": {"logicalType": "timestamp-millis", "type": "long"}}], "name": "example", "type": "record"}
{code}
if you really want to call {{json.dumps}} on a schema instance directly, you need to provide the MappingProxyEncoder:
{code:python}
>>> print(json.dumps(example.to_json(), cls=avro.schema.MappingProxyEncoder, indent=4))
{
"fields": [
{
"name": "exfield",
"type": {
"logicalType": "timestamp-millis",
"type": "long"
}
}
],
"name": "example",
"type": "record"
}
{code}
was (Author: kojiromike):
I see. This was caused by the changes in AVRO-2234. Sorry for the inconvenience.
If you call {{json.dumps}} directly on a schema or on the output of {{Schema.to_json()}}, you'll see errors like you describe. But you probably should not call {{json.dumps}} directly on a schema instance. The {{__str__}} implementation of a schema object is exactly {{json.dumps}}, except that it includes an implementation of {{MappingProxyEncoder}} to solve this problem.
{code:python}
>>> import avro.schema, json, pprint
>>> example = avro.schema.parse('{"type":"record","name":"example","fields":[{"name":"exfield","type":{"type":"long","logicalType":"timestamp-millis"}}]}')
>>> schema_dict = example.to_json()
>>> pprint.pprint(schema_dict)
{'fields': [{'name': 'exfield',
'type': mappingproxy({'logicalType': 'timestamp-millis',
'type': 'long'})}],
'name': 'example',
'type': 'record'}
>>> json.dumps(example)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.5/json/__init__.py", line 230, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python3.5/json/encoder.py", line 198, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python3.5/json/encoder.py", line 256, in iterencode
return _iterencode(o, 0)
File "/usr/lib/python3.5/json/encoder.py", line 179, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <avro.schema.RecordSchema object at 0x7fcecdd53518> is not JSON serializable
{code}
...but
{code:python}
>>> print(str(example))
{"fields": [{"name": "exfield", "type": {"logicalType": "timestamp-millis", "type": "long"}}], "name": "example", "type": "record"}
{code:python}
if you really want to call {{json.dumps}} on a schema instance directly, you need to provide the MappingProxyEncoder:
{code:python}
>>> print(json.dumps(example.to_json(), cls=avro.schema.MappingProxyEncoder, indent=4))
{
"fields": [
{
"name": "exfield",
"type": {
"logicalType": "timestamp-millis",
"type": "long"
}
}
],
"name": "example",
"type": "record"
}
{code}
> RecordSchema with logicalType field is not JSON serializable
> ------------------------------------------------------------
>
> Key: AVRO-2411
> URL: https://issues.apache.org/jira/browse/AVRO-2411
> Project: Apache Avro
> Issue Type: Bug
> Components: python
> Affects Versions: 1.9.0
> Reporter: Jonathan
> Assignee: Michael A. Smith
> Priority: Major
>
> After updating to 1.9.0, RecordSchemas that contain a field with a logicalType mapping can no longer be passed into json.dumps() as it throws the following error:
> {code}Object of type mappingproxy is not JSON serializable{code}
> Example schema:
> {code}
> {
> "namespace": "Sherlock",
> "type": "record",
> "name": "Action",
> "fields": [
> {
> "name": "action_id",
> "type": "string",
> "doc": "Key for the Action"
> },
> {
> "name": "actionType_id",
> "type": "int",
> "doc": "Action Type key"
> },
> {
> "name": "timestamp",
> "type": {
> "type": "long",
> "logicalType": "timestamp-millis"
> },
> "doc": "Event time of validation request or modified_dt of save (UTC)"
> }
> ]
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)