You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fury.apache.org by ch...@apache.org on 2024/05/06 14:56:31 UTC
(incubator-fury) branch main updated: feat(sepc): update type meta field info spec (#1603)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new 93800888 feat(sepc): update type meta field info spec (#1603)
93800888 is described below
commit 93800888595065b2690fec093ab0cbfd6ac7dedc
Author: Shawn Yang <sh...@gmail.com>
AuthorDate: Mon May 6 22:56:27 2024 +0800
feat(sepc): update type meta field info spec (#1603)
## What does this PR do?
Update type meta field info spec:
```
- field info:
- header(8
bits): `3 bits size + 2 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag`.
Users can use annotation to provide those info.
- 2 bits field name encoding:
- encoding: `UTF8/ALL_TO_LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL/TAG_ID`
- If tag id is used, i.e. field name is written by an unsigned varint tag id. 2 bits encoding will be `11`.
- size of field name:
- The `3 bits size: 0~7` will be used to indicate length `1~7`, the value `7` indicates to read more bytes,
the encoding will encode `size - 7` as a varint next.
- If encoding is `TAG_ID`, then num_bytes of field name will be used to store tag id.
- ref tracking: when set to 1, ref tracking will be enabled for this field.
- nullability: when set to 1, this field can be null.
- polymorphism: when set to 1, the actual type of field will be the declared field type even the type if
not `final`.
- field name: If tag id is set, tag id will be used instead. Otherwise meta string encoding `[length]` and data will
be written instead.
```
## Related issues
#1556
## Does this PR introduce any user-facing change?
<!--
If any user-facing interface changes, please [open an
issue](https://github.com/apache/incubator-fury/issues/new/choose)
describing the need to do so and update the document if necessary.
-->
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
<!--
When the PR has an impact on performance (if you don't know whether the
PR will have an impact on performance, you can submit the PR first, and
if it will have impact on performance, the code reviewer will explain
it), be sure to attach a benchmark data here.
-->
---
docs/specification/xlang_serialization_spec.md | 34 ++++++++++++++------------
1 file changed, 19 insertions(+), 15 deletions(-)
diff --git a/docs/specification/xlang_serialization_spec.md b/docs/specification/xlang_serialization_spec.md
index 0583e7e0..7b46b556 100644
--- a/docs/specification/xlang_serialization_spec.md
+++ b/docs/specification/xlang_serialization_spec.md
@@ -288,23 +288,27 @@ Meta header is a 64 bits number value encoded in little endian order.
fields, then use fields info in meta for deserializing compatible fields.
- type id: the registered id for the current type, which will be written as an unsigned varint.
- field info:
- - Header(8 bits):
- - Format:
- - `reserved 1 bit + 3 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag + tag id flag`.
- - Users can use annotation to provide that info.
- - tag id: when set to 1, the field name will be written by an unsigned varint tag id.
- - ref tracking: when set to 0, ref tracking will be disabled for this field.
- - nullability: when set to 0, this field won't be null.
- - polymorphism: when set to 1, the actual type of field will be the declared field type even the type if
- not `final`.
- - 3 bits field name encoding will be set to meta string encoding flags when tag id is not set.
- - Type id:
+ - header(8
+ bits): `3 bits size + 2 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag`.
+ Users can use annotation to provide those info.
+ - 2 bits field name encoding:
+ - encoding: `UTF8/ALL_TO_LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL/TAG_ID`
+ - If tag id is used, i.e. field name is written by an unsigned varint tag id. 2 bits encoding will be `11`.
+ - size of field name:
+ - The `3 bits size: 0~7` will be used to indicate length `1~7`, the value `7` indicates to read more bytes,
+ the encoding will encode `size - 7` as a varint next.
+ - If encoding is `TAG_ID`, then num_bytes of field name will be used to store tag id.
+ - ref tracking: when set to 1, ref tracking will be enabled for this field.
+ - nullability: when set to 1, this field can be null.
+ - polymorphism: when set to 1, the actual type of field will be the declared field type even the type if
+ not `final`.
+ - field name: If tag id is set, tag id will be used instead. Otherwise meta string encoding `[length]` and data will
+ be written instead.
+ - type id:
- For registered type-consistent classes, it will be the registered type id.
- Otherwise it will be encoded as `OBJECT_ID` if it isn't `final` and `FINAL_OBJECT_ID` if it's `final`. The
meta for such types is written separately instead of inlining here is to reduce meta space cost if object of
- this type is serialized in the current object graph multiple times, and the field value may be null too.
- - Field name: If tag id is set, tag id will be used instead. Otherwise meta string encoding length and data will
- be written instead.
+ this type is serialized in current object graph multiple times, and the field value may be null too.
Field order are left as implementation details, which is not exposed to specification, the deserialization need to
resort fields based on Fury field comparator. In this way, fury can compute statistics for field names or types and
@@ -473,7 +477,7 @@ which will be encoded by elements header, each use one bit:
By default, all bits are unset, which means all elements won't track ref, all elements are same type, not null and
the actual element is the declared type in the custom type field.
-The implementation can generate different deserialization code based read header, and look up the generated code from
+The implementation can generate different deserialization code based read header, and look up the generated code from
a linear map/list.
#### elements data
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org