You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fury.apache.org by ch...@apache.org on 2024/08/17 15:03:13 UTC
(fury-site) branch main updated (5d3335b -> 7938c1f)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/fury-site.git
from 5d3335b Docs:add Chinese and English documentation on how to join the fury community. (#142)
new 68f3405 🔄 synced local 'docs/guide/' with remote 'docs/guide/'
new 752aab9 🔄 synced local 'docs/specification/' with remote 'docs/specification/'
new 7938c1f 🔄 synced local 'static/img/benchmarks/' with remote 'docs/benchmarks/'
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/guide/DEVELOPMENT.md | 5 +-
docs/guide/graalvm_guide.md | 27 +--
docs/guide/java_serialization_guide.md | 10 +-
docs/guide/row_format_guide.md | 12 --
docs/guide/scala_guide.md | 21 +--
docs/guide/xlang_serialization_guide.md | 239 ++++++++++++-------------
docs/guide/xlang_type_mapping.md | 6 +-
docs/specification/java_serialization_spec.md | 72 ++++----
docs/specification/row_format_spec.md | 1 +
docs/specification/xlang_serialization_spec.md | 112 ++++++------
static/img/benchmarks/README.md | 34 +---
11 files changed, 227 insertions(+), 312 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
(fury-site) 03/03: 🔄 synced local 'static/img/benchmarks/' with remote 'docs/benchmarks/'
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/fury-site.git
commit 7938c1f4e3ddd23055ce7e91dda411fb42333ecc
Author: chaokunyang <sh...@gmail.com>
AuthorDate: Sat Aug 17 15:03:08 2024 +0000
🔄 synced local 'static/img/benchmarks/' with remote 'docs/benchmarks/'
---
static/img/benchmarks/README.md | 34 +---------------------------------
1 file changed, 1 insertion(+), 33 deletions(-)
diff --git a/static/img/benchmarks/README.md b/static/img/benchmarks/README.md
index d380e51..1bc562a 100644
--- a/static/img/benchmarks/README.md
+++ b/static/img/benchmarks/README.md
@@ -1,7 +1,5 @@
# Java Benchmarks
-
## System Environment
-
- Operation System:4.9.151-015.x86_64
- CPU:Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
- Byte Order:Little Endian
@@ -11,19 +9,14 @@
- L3 cache: 33792K
## JMH params
-
Don't skip **warm up**, otherwise the results aren't accurate.
-
```bash
-f 1 -wi 3 -i 3 -t 1 -w 2s -r 2s -rf cs
```
## Benchmark Data:
-
### Struct
-
Struct is a class with 100 primitive fields:
-
```java
public class Struct {
public int f1;
@@ -34,11 +27,8 @@ public class Struct {
public double f99;
}
```
-
### Struct2
-
Struct2 is a class with 100 boxed fields:
-
```java
public class Struct {
public Integer f1;
@@ -49,23 +39,16 @@ public class Struct {
public Double f99;
}
```
-
### MediaContent
-
MEDIA_CONTENT is a class from [jvm-serializers](https://github.com/eishay/jvm-serializers/blob/master/tpc/src/data/media/MediaContent.java).
-
### Sample
-
SAMPLE is a class from [kryo benchmark](https://github.com/EsotericSoftware/kryo/blob/master/benchmarks/src/main/java/com/esotericsoftware/kryo/benchmarks/data/Sample.java)
## Benchmark Plots
-
### Serialize to heap buffer
-
Serialize data java byte array.
#### Java schema consistent serialization
-
The deserialization peer must have same class definition with the serialization peer.
No class forward/backward compatibility are supported in this mode.
@@ -77,7 +60,6 @@ No class forward/backward compatibility are supported in this mode.
</p>
#### Java schema compatible serialization
-
The deserialization peer can have different class definition with the serialization peer.
Class forward/backward compatibility are supported in this mode.
@@ -89,7 +71,6 @@ Class forward/backward compatibility are supported in this mode.
</p>
#### Java schema consistent deserialization
-
The deserialization peer must have same class definition with the serialization peer.
No class forward/backward compatibility are supported in this mode.
@@ -101,7 +82,6 @@ No class forward/backward compatibility are supported in this mode.
</p>
#### Java schema compatible deserialization
-
The deserialization peer can have different class definition with the serialization peer.
Class forward/backward compatibility are supported in this mode.
<p align="center">
@@ -112,11 +92,9 @@ Class forward/backward compatibility are supported in this mode.
</p>
### Off-heap serialization
-
Serialize data off-heap memory.
#### Java schema consistent serialization
-
The deserialization peer must have same class definition with the serialization peer.
No class forward/backward compatibility are supported in this mode.
<p align="center">
@@ -127,7 +105,6 @@ No class forward/backward compatibility are supported in this mode.
</p>
#### Java schema compatible serialization
-
The deserialization peer can have different class definition with the serialization peer.
Class forward/backward compatibility are supported in this mode.
<p align="center">
@@ -138,7 +115,6 @@ Class forward/backward compatibility are supported in this mode.
</p>
#### Java schema consistent deserialization
-
The deserialization peer must have same class definition with the serialization peer.
No class forward/backward compatibility are supported in this mode.
<p align="center">
@@ -149,7 +125,6 @@ No class forward/backward compatibility are supported in this mode.
</p>
#### Java schema compatible deserialization
-
The deserialization peer can have different class definition with the serialization peer.
Class forward/backward compatibility are supported in this mode.
<p align="center">
@@ -160,13 +135,10 @@ Class forward/backward compatibility are supported in this mode.
</p>
### Zero-copy serialization
-
-Note that zero-copy serialization just avoid the copy in serialization, if you send data to other machine, there may be copies.
+Note that zero-copy serialization just avoid the copy in serialization, if you send data to other machine, there may be copies.
But if you serialize data between processes on same node and use shared-memory, if the data are in off-heap before serialization, then other processes can read this buffer without any copies.
-
#### Java zero-copy serialize to heap buffer
-
<p align="center">
<img width="24%" alt="" src="zerocopy/zero_copy_bench_serialize_BUFFER_to_array_tps.png">
<img width="24%" alt="" src="zerocopy/zero_copy_bench_serialize_BUFFER_to_directBuffer_tps.png">
@@ -175,7 +147,6 @@ But if you serialize data between processes on same node and use shared-memory,
</p>
#### Java zero-copy serialize to direct buffer
-
<p align="center">
<img width="24%" alt="" src="zerocopy/zero_copy_bench_deserialize_BUFFER_from_array_tps.png">
<img width="24%" alt="" src="zerocopy/zero_copy_bench_deserialize_BUFFER_from_directBuffer_tps.png">
@@ -184,9 +155,7 @@ But if you serialize data between processes on same node and use shared-memory,
</p>
## Benchmark Data
-
### Java Serialization
-
| Lib | Benchmark | bufferType | objectType | references | Tps |
| ------- | ------- | ------- | ------- | ------- | ------- |
| Fst | serialize | array | SAMPLE | False | 915907.574306 |
@@ -495,7 +464,6 @@ But if you serialize data between processes on same node and use shared-memory,
| Protostuff | deserialize | directBuffer | STRUCT2 | False | 425523.315814 |
### Java Zero-copy
-
| Lib | Benchmark | array_size | bufferType | dataType | Tps |
| ------- | ------- | ------- | ------- | ------- | ------- |
| Fst | deserialize | 200 | array | PRIMITIVE_ARRAY | 219333.990504 |
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
(fury-site) 01/03: 🔄 synced local 'docs/guide/' with remote 'docs/guide/'
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/fury-site.git
commit 68f34059d26358f8f5abfbfffccd0542b43863db
Author: chaokunyang <sh...@gmail.com>
AuthorDate: Sat Aug 17 15:03:08 2024 +0000
🔄 synced local 'docs/guide/' with remote 'docs/guide/'
---
docs/guide/DEVELOPMENT.md | 5 +-
docs/guide/graalvm_guide.md | 27 +---
docs/guide/java_serialization_guide.md | 10 +-
docs/guide/row_format_guide.md | 12 --
docs/guide/scala_guide.md | 21 +--
docs/guide/xlang_serialization_guide.md | 239 ++++++++++++++++----------------
docs/guide/xlang_type_mapping.md | 6 +-
7 files changed, 135 insertions(+), 185 deletions(-)
diff --git a/docs/guide/DEVELOPMENT.md b/docs/guide/DEVELOPMENT.md
index 39d8605..3949dcb 100644
--- a/docs/guide/DEVELOPMENT.md
+++ b/docs/guide/DEVELOPMENT.md
@@ -1,9 +1,11 @@
---
-title: How to build Fury
+title: Development
sidebar_position: 7
id: development
---
+# How to build Fury
+
Please checkout the source tree from https://github.com/apache/fury.
### Build Fury Java
@@ -97,3 +99,4 @@ npm run test
- node 14+
- npm 8+
+
diff --git a/docs/guide/graalvm_guide.md b/docs/guide/graalvm_guide.md
index 0333a43..3ed919f 100644
--- a/docs/guide/graalvm_guide.md
+++ b/docs/guide/graalvm_guide.md
@@ -4,6 +4,7 @@ sidebar_position: 6
id: graalvm_guide
---
+# GraalVM Native Image
GraalVM `native image` can compile java code into native code ahead to build faster, smaller, leaner applications.
The native image doesn't have a JIT compiler to compile bytecode into machine code, and doesn't support
reflection unless configure reflection file.
@@ -15,7 +16,6 @@ In order to use Fury on graalvm native image, you must create Fury as an **stati
the enclosing class initialize time. Then configure `native-image.properties` under
`resources/META-INF/native-image/$xxx/native-image.propertie` to tell graalvm to init the class at native image
build time. For example, here we configure `org.apache.fury.graalvm.Example` class be init at build time:
-
```properties
Args = --initialize-at-build-time=org.apache.fury.graalvm.Example
```
@@ -29,9 +29,7 @@ Note that Fury `asyncCompilationEnabled` option will be disabled automatically f
native image doesn't support JIT at the image run time.
## Not thread-safe Fury
-
Example:
-
```java
import org.apache.fury.Fury;
import org.apache.fury.util.Preconditions;
@@ -65,15 +63,12 @@ public class Example {
}
}
```
-
Then add `org.apache.fury.graalvm.Example` build time init to `native-image.properties` configuration:
-
```properties
Args = --initialize-at-build-time=org.apache.fury.graalvm.Example
```
## Thread-safe Fury
-
```java
import org.apache.fury.Fury;
import org.apache.fury.ThreadLocalFury;
@@ -114,40 +109,32 @@ public class ThreadSafeExample {
}
}
```
-
Then add `org.apache.fury.graalvm.ThreadSafeExample` build time init to `native-image.properties` configuration:
-
```properties
Args = --initialize-at-build-time=org.apache.fury.graalvm.ThreadSafeExample
```
## Framework Integration
-
For framework developers, if you want to integrate fury for serialization, you can provided a configuration file to let
the users to list all the classes they want to serialize, then you can load those classes and invoke
`org.apache.fury.Fury.register(Class<?>, boolean)` to register those classes in your Fury integration class, and configure that
class be initialized at graalvm native image build time.
## Benchmark
-
Here we give two class benchmarks between Fury and Graalvm Serialization.
When Fury compression is disabled:
-
- Struct: Fury is `46x speed, 43% size` compared to JDK.
- Pojo: Fury is `12x speed, 56% size` compared to JDK.
When Fury compression is enabled:
-
- Struct: Fury is `24x speed, 31% size` compared to JDK.
- Pojo: Fury is `12x speed, 48% size` compared to JDK.
See [[Benchmark.java](https://github.com/apache/fury/blob/main/integration_tests/graalvm_tests/src/main/java/org/apache/fury/graalvm/Benchmark.java)] for benchmark code.
### Struct Benchmark
-
#### Class Fields
-
```java
public class Struct implements Serializable {
public int f1;
@@ -164,11 +151,8 @@ public class Struct implements Serializable {
public double f12;
}
```
-
#### Benchmark Results
-
No compression:
-
```
Benchmark repeat number: 400000
Object type: class org.apache.fury.graalvm.Struct
@@ -180,9 +164,7 @@ JDK serialization took mills: 2254
Compare speed: Fury is 45.70x speed of JDK
Compare size: Fury is 0.43x size of JDK
```
-
Compress number:
-
```
Benchmark repeat number: 400000
Object type: class org.apache.fury.graalvm.Struct
@@ -196,9 +178,7 @@ Compare size: Fury is 0.31x size of JDK
```
### Pojo Benchmark
-
#### Class Fields
-
```java
public class Foo implements Serializable {
int f1;
@@ -207,11 +187,8 @@ public class Foo implements Serializable {
Map<String, Long> f4;
}
```
-
#### Benchmark Results
-
No compression:
-
```
Benchmark repeat number: 400000
Object type: class org.apache.fury.graalvm.Foo
@@ -223,9 +200,7 @@ JDK serialization took mills: 16266
Compare speed: Fury is 12.19x speed of JDK
Compare size: Fury is 0.56x size of JDK
```
-
Compress number:
-
```
Benchmark repeat number: 400000
Object type: class org.apache.fury.graalvm.Foo
diff --git a/docs/guide/java_serialization_guide.md b/docs/guide/java_serialization_guide.md
index e29b0b2..de179f4 100644
--- a/docs/guide/java_serialization_guide.md
+++ b/docs/guide/java_serialization_guide.md
@@ -4,6 +4,8 @@ sidebar_position: 0
id: java_object_graph_guide
---
+# Java object graph serialization
+
When only java object serialization needed, this mode will have better performance compared to cross-language object
graph serialization.
@@ -177,12 +179,12 @@ bit is set, then next byte will be read util first bit of next byte is unset.
For long compression, fury support two encoding:
- Fury SLI(Small long as int) Encoding (**used by default**):
- - If long is in [-1073741824, 1073741823], encode as 4 bytes int: `| little-endian: ((int) value) << 1 |`
- - Otherwise write as 9 bytes: `| 0b1 | little-endian 8bytes long |`
+ - If long is in [-1073741824, 1073741823], encode as 4 bytes int: `| little-endian: ((int) value) << 1 |`
+ - Otherwise write as 9 bytes: `| 0b1 | little-endian 8bytes long |`
- Fury PVL(Progressive Variable-length Long) Encoding:
- - First bit in every byte indicate whether has next byte. if first bit is set, then next byte will be read util
+ - First bit in every byte indicate whether has next byte. if first bit is set, then next byte will be read util
first bit of next byte is unset.
- - Negative number will be converted to positive number by `(v << 1) ^ (v >> 63)` to reduce cost of small negative
+ - Negative number will be converted to positive number by ` (v << 1) ^ (v >> 63)` to reduce cost of small negative
numbers.
If a number are `long` type, it can't be represented by smaller bytes mostly, the compression won't get good enough
diff --git a/docs/guide/row_format_guide.md b/docs/guide/row_format_guide.md
index 1083297..076a2c0 100644
--- a/docs/guide/row_format_guide.md
+++ b/docs/guide/row_format_guide.md
@@ -5,9 +5,7 @@ id: row_format_guide
---
## Row format protocol
-
### Java
-
```java
public class Bar {
String f1;
@@ -52,9 +50,7 @@ RowEncoder<Bar> barEncoder = Encoders.bean(Bar.class);
Bar newBar = barEncoder.fromRow(barStruct);
Bar newBar2 = barEncoder.fromRow(binaryArray4.getStruct(20));
```
-
### Python
-
```python
@dataclass
class Bar:
@@ -83,13 +79,10 @@ new_foo = pickle.loads(binary)
print(new_foo.f2[100000], new_foo.f4[100000].f1, new_foo.f4[200000].f2[5])
print(f"pickle end: {datetime.datetime.now()}")
```
-
### Apache Arrow Support
-
Fury Format also supports automatic conversion from/to Arrow Table/RecordBatch.
Java:
-
```java
Schema schema = TypeInference.inferSchema(BeanA.class);
ArrowWriter arrowWriter = ArrowUtils.createArrowWriter(schema);
@@ -100,18 +93,14 @@ for (int i = 0; i < 10; i++) {
}
return arrowWriter.finishAsRecordBatch();
```
-
Python:
-
```python
import pyfury
encoder = pyfury.encoder(Foo)
encoder.to_arrow_record_batch([foo] * 10000)
encoder.to_arrow_table([foo] * 10000)
```
-
C++
-
```c++
std::shared_ptr<ArrowWriter> arrow_writer;
EXPECT_TRUE(
@@ -126,7 +115,6 @@ EXPECT_TRUE(record_batch->Validate().ok());
EXPECT_EQ(record_batch->num_columns(), schema->num_fields());
EXPECT_EQ(record_batch->num_rows(), row_nums);
```
-
```java
Schema schema = TypeInference.inferSchema(BeanA.class);
ArrowWriter arrowWriter = ArrowUtils.createArrowWriter(schema);
diff --git a/docs/guide/scala_guide.md b/docs/guide/scala_guide.md
index 4de2f09..aa8f99b 100644
--- a/docs/guide/scala_guide.md
+++ b/docs/guide/scala_guide.md
@@ -4,8 +4,8 @@ sidebar_position: 4
id: scala_guide
---
+# Scala serialization
Fury supports all scala object serialization:
-
- `case` class serialization supported
- `pojo/bean` class serialization supported
- `object` singleton serialization supported
@@ -15,15 +15,12 @@ Fury supports all scala object serialization:
Scala 2 and 3 are both supported.
## Install
-
```sbt
libraryDependencies += "org.apache.fury" % "fury-core" % "0.7.0"
```
## Fury creation
-
When using fury for scala serialization, you should create fury at least with following options:
-
```scala
val fury = Fury.builder()
.withScalaOptimizationEnabled(true)
@@ -31,14 +28,11 @@ val fury = Fury.builder()
.withRefTracking(true)
.build()
```
-
Depending on the object types you serialize, you may need to register some scala internal types:
-
```scala
fury.register(Class.forName("scala.collection.generic.DefaultSerializationProxy"))
fury.register(Class.forName("scala.Enumeration.Val"))
```
-
If you want to avoid such registration, you can disable class registration by `FuryBuilder#requireClassRegistration(false)`.
Note that this option allow to deserialize objects unknown types, more flexible but may be insecure if the classes contains malicious code.
@@ -49,7 +43,6 @@ Note that fury instance should be shared between multiple serialization, the cre
If you use shared fury instance across multiple threads, you should create `ThreadSafeFury` instead by `FuryBuilder#buildThreadSafeFury()` instead.
## Serialize case object
-
```scala
case class Person(github: String, age: Int, id: Long)
val p = Person("https://github.com/chaokunyang", 18, 1)
@@ -58,7 +51,6 @@ println(fury.deserializeJavaObject(fury.serializeJavaObject(p)))
```
## Serialize pojo
-
```scala
class Foo(f1: Int, f2: String) {
override def toString: String = s"Foo($f1, $f2)"
@@ -67,7 +59,6 @@ println(fury.deserialize(fury.serialize(Foo(1, "chaokunyang"))))
```
## Serialize object singleton
-
```scala
object singleton {
}
@@ -77,7 +68,6 @@ println(o1 == o2)
```
## Serialize collection
-
```scala
val seq = Seq(1,2)
val list = List("a", "b")
@@ -88,7 +78,6 @@ println(fury.deserialize(fury.serialize(map)))
```
## Serialize Tuple
-
```scala
val tuple = Tuple2(100, 10000L)
println(fury.deserialize(fury.serialize(tuple)))
@@ -97,16 +86,12 @@ println(fury.deserialize(fury.serialize(tuple)))
```
## Serialize Enum
-
### Scala3 Enum
-
```scala
enum Color { case Red, Green, Blue }
println(fury.deserialize(fury.serialize(Color.Green)))
```
-
### Scala2 Enum
-
```scala
object ColorEnum extends Enumeration {
type ColorEnum = Value
@@ -116,7 +101,6 @@ println(fury.deserialize(fury.serialize(ColorEnum.Green)))
```
## Serialize Option
-
```scala
val opt: Option[Long] = Some(100)
println(fury.deserialize(fury.serialize(opt)))
@@ -124,8 +108,7 @@ val opt1: Option[Long] = None
println(fury.deserialize(fury.serialize(opt1)))
```
-## Performance
-
+# Performance
Scala `pojo/bean/case/object` are supported by fury jit well, the performance is as good as fury java.
Scala collections and generics doesn't follow java collection framework, and is not fully integrated with Fury JIT in current release version. The performance won't be as good as fury collections serialization for java.
diff --git a/docs/guide/xlang_serialization_guide.md b/docs/guide/xlang_serialization_guide.md
index 5bc6a05..a68348e 100644
--- a/docs/guide/xlang_serialization_guide.md
+++ b/docs/guide/xlang_serialization_guide.md
@@ -7,9 +7,9 @@ id: xlang_object_graph_guide
## Cross-language object graph serialization
### Serialize built-in types
-
Common types can be serialized automatically: primitive numeric types, string, binary, array, list, map and so on.
+
**Java**
```java
@@ -64,32 +64,32 @@ import furygo "github.com/apache/fury/fury/go/fury"
import "fmt"
func main() {
- list := []interface{}{true, false, "str", -1.1, 1, make([]int32, 10), make([]float64, 20)}
- fury := furygo.NewFury()
- bytes, err := fury.Marshal(list)
- if err != nil {
- panic(err)
- }
- var newValue interface{}
- // bytes can be data serialized by other languages.
- if err := fury.Unmarshal(bytes, &newValue); err != nil {
- panic(err)
- }
- fmt.Println(newValue)
- dict := map[string]interface{}{
- "k1": "v1",
- "k2": list,
- "k3": -1,
- }
- bytes, err = fury.Marshal(dict)
- if err != nil {
- panic(err)
- }
- // bytes can be data serialized by other languages.
- if err := fury.Unmarshal(bytes, &newValue); err != nil {
- panic(err)
- }
- fmt.Println(newValue)
+ list := []interface{}{true, false, "str", -1.1, 1, make([]int32, 10), make([]float64, 20)}
+ fury := furygo.NewFury()
+ bytes, err := fury.Marshal(list)
+ if err != nil {
+ panic(err)
+ }
+ var newValue interface{}
+ // bytes can be data serialized by other languages.
+ if err := fury.Unmarshal(bytes, &newValue); err != nil {
+ panic(err)
+ }
+ fmt.Println(newValue)
+ dict := map[string]interface{}{
+ "k1": "v1",
+ "k2": list,
+ "k3": -1,
+ }
+ bytes, err = fury.Marshal(dict)
+ if err != nil {
+ panic(err)
+ }
+ // bytes can be data serialized by other languages.
+ if err := fury.Unmarshal(bytes, &newValue); err != nil {
+ panic(err)
+ }
+ fmt.Println(newValue)
}
```
@@ -126,7 +126,6 @@ fn run() {
```
### Serialize custom types
-
Serializing user-defined types needs registering the custom type using the register API to establish the mapping relationship between the type in different languages.
**Java**
@@ -256,59 +255,59 @@ import furygo "github.com/apache/fury/fury/go/fury"
import "fmt"
func main() {
- type SomeClass1 struct {
- F1 interface{}
- F2 string
- F3 []interface{}
- F4 map[int8]int32
- F5 int8
- F6 int16
- F7 int32
- F8 int64
- F9 float32
- F10 float64
- F11 []int16
- F12 fury.Int16Slice
- }
-
- type SomeClas2 struct {
- F1 interface{}
- F2 map[int8]int32
- }
- fury := furygo.NewFury()
- if err := fury.RegisterTagType("example.SomeClass1", SomeClass1{}); err != nil {
- panic(err)
- }
- if err := fury.RegisterTagType("example.SomeClass2", SomeClass2{}); err != nil {
- panic(err)
- }
- obj1 := &SomeClass1{}
- obj1.F1 = true
- obj1.F2 = map[int8]int32{-1: 2}
- obj := &SomeClass1{}
- obj.F1 = obj1
- obj.F2 = "abc"
- obj.F3 = []interface{}{"abc", "abc"}
- f4 := map[int8]int32{1: 2}
- obj.F4 = f4
- obj.F5 = fury.MaxInt8
- obj.F6 = fury.MaxInt16
- obj.F7 = fury.MaxInt32
- obj.F8 = fury.MaxInt64
- obj.F9 = 1.0 / 2
- obj.F10 = 1 / 3.0
- obj.F11 = []int16{1, 2}
- obj.F12 = []int16{-1, 4}
- bytes, err := fury.Marshal(obj);
- if err != nil {
- panic(err)
- }
- var newValue interface{}
- // bytes can be data serialized by other languages.
- if err := fury.Unmarshal(bytes, &newValue); err != nil {
- panic(err)
- }
- fmt.Println(newValue)
+ type SomeClass1 struct {
+ F1 interface{}
+ F2 string
+ F3 []interface{}
+ F4 map[int8]int32
+ F5 int8
+ F6 int16
+ F7 int32
+ F8 int64
+ F9 float32
+ F10 float64
+ F11 []int16
+ F12 fury.Int16Slice
+ }
+
+ type SomeClas2 struct {
+ F1 interface{}
+ F2 map[int8]int32
+ }
+ fury := furygo.NewFury()
+ if err := fury.RegisterTagType("example.SomeClass1", SomeClass1{}); err != nil {
+ panic(err)
+ }
+ if err := fury.RegisterTagType("example.SomeClass2", SomeClass2{}); err != nil {
+ panic(err)
+ }
+ obj1 := &SomeClass1{}
+ obj1.F1 = true
+ obj1.F2 = map[int8]int32{-1: 2}
+ obj := &SomeClass1{}
+ obj.F1 = obj1
+ obj.F2 = "abc"
+ obj.F3 = []interface{}{"abc", "abc"}
+ f4 := map[int8]int32{1: 2}
+ obj.F4 = f4
+ obj.F5 = fury.MaxInt8
+ obj.F6 = fury.MaxInt16
+ obj.F7 = fury.MaxInt32
+ obj.F8 = fury.MaxInt64
+ obj.F9 = 1.0 / 2
+ obj.F10 = 1 / 3.0
+ obj.F11 = []int16{1, 2}
+ obj.F12 = []int16{-1, 4}
+ bytes, err := fury.Marshal(obj);
+ if err != nil {
+ panic(err)
+ }
+ var newValue interface{}
+ // bytes can be data serialized by other languages.
+ if err := fury.Unmarshal(bytes, &newValue); err != nil {
+ panic(err)
+ }
+ fmt.Println(newValue)
}
```
@@ -395,7 +394,6 @@ fn complex_struct() {
```
### Serialize Shared Reference and Circular Reference
-
Shared reference and circular reference can be serialized automatically, no duplicate data or recursion error.
**Java**
@@ -462,27 +460,27 @@ import furygo "github.com/apache/fury/fury/go/fury"
import "fmt"
func main() {
- type SomeClass struct {
- F1 *SomeClass
- F2 map[string]string
- F3 map[string]string
- }
- fury := furygo.NewFury(true)
- if err := fury.RegisterTagType("example.SomeClass", SomeClass{}); err != nil {
- panic(err)
- }
- value := &SomeClass{F2: map[string]string{"k1": "v1", "k2": "v2"}}
- value.F3 = value.F2
- value.F1 = value
- bytes, err := fury.Marshal(value)
- if err != nil {
- }
- var newValue interface{}
- // bytes can be data serialized by other languages.
- if err := fury.Unmarshal(bytes, &newValue); err != nil {
- panic(err)
- }
- fmt.Println(newValue)
+ type SomeClass struct {
+ F1 *SomeClass
+ F2 map[string]string
+ F3 map[string]string
+ }
+ fury := furygo.NewFury(true)
+ if err := fury.RegisterTagType("example.SomeClass", SomeClass{}); err != nil {
+ panic(err)
+ }
+ value := &SomeClass{F2: map[string]string{"k1": "v1", "k2": "v2"}}
+ value.F3 = value.F2
+ value.F1 = value
+ bytes, err := fury.Marshal(value)
+ if err != nil {
+ }
+ var newValue interface{}
+ // bytes can be data serialized by other languages.
+ if err := fury.Unmarshal(bytes, &newValue); err != nil {
+ panic(err)
+ }
+ fmt.Println(newValue)
}
```
@@ -516,6 +514,7 @@ console.log(result.bar.foo === result.foo);
**JavaScript**
Reference cannot be implemented because of rust ownership restrictions
+
### Zero-Copy Serialization
**Java**
@@ -570,23 +569,23 @@ import furygo "github.com/apache/fury/fury/go/fury"
import "fmt"
func main() {
- fury := furygo.NewFury()
- list := []interface{}{"str", make([]byte, 1000)}
- buf := fury.NewByteBuffer(nil)
- var bufferObjects []fury.BufferObject
- fury.Serialize(buf, list, func(o fury.BufferObject) bool {
- bufferObjects = append(bufferObjects, o)
- return false
- })
- var newList []interface{}
- var buffers []*fury.ByteBuffer
- for _, o := range bufferObjects {
- buffers = append(buffers, o.ToBuffer())
- }
- if err := fury.Deserialize(buf, &newList, buffers); err != nil {
- panic(err)
- }
- fmt.Println(newList)
+ fury := furygo.NewFury()
+ list := []interface{}{"str", make([]byte, 1000)}
+ buf := fury.NewByteBuffer(nil)
+ var bufferObjects []fury.BufferObject
+ fury.Serialize(buf, list, func(o fury.BufferObject) bool {
+ bufferObjects = append(bufferObjects, o)
+ return false
+ })
+ var newList []interface{}
+ var buffers []*fury.ByteBuffer
+ for _, o := range bufferObjects {
+ buffers = append(buffers, o.ToBuffer())
+ }
+ if err := fury.Deserialize(buf, &newList, buffers); err != nil {
+ panic(err)
+ }
+ fmt.Println(newList)
}
```
diff --git a/docs/guide/xlang_type_mapping.md b/docs/guide/xlang_type_mapping.md
index 7be6900..4baa455 100644
--- a/docs/guide/xlang_type_mapping.md
+++ b/docs/guide/xlang_type_mapping.md
@@ -10,6 +10,8 @@ Note:
- `int16_t[n]/vector<T>` indicates `int16_t[n]/vector<int16_t>`
- The cross-language serialization is not stable, do not use it in your production environment.
+# Type Mapping
+
| Fury Type | Fury Type ID | Java | Python | Javascript | C++ | Golang | Rust |
|--------------------|--------------|-----------------|----------------------|-----------------|--------------------------------|------------------|------------------|
| bool | 1 | bool/Boolean | bool | Boolean | bool | bool | bool |
@@ -46,7 +48,7 @@ Note:
| arrow record batch | 32 | / | / | / | / | / | / |
| arrow table | 33 | / | / | / | / | / | / |
-## Type info(not implemented currently)
+# Type info(not implemented currently)
Due to differences between type systems of languages, those types can't be mapped one-to-one between languages.
@@ -68,7 +70,6 @@ Such information can be provided in other languages too:
Here is en example:
- Java:
-
```java
class Foo {
@Int32Type(varint = true)
@@ -76,7 +77,6 @@ Here is en example:
List<@Int32Type(varint = true) Integer> f2;
}
```
-
- Python:
```python
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
(fury-site) 02/03: 🔄 synced local 'docs/specification/' with remote 'docs/specification/'
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/fury-site.git
commit 752aab97d9a0d099270ad14d285b662c72d0656e
Author: chaokunyang <sh...@gmail.com>
AuthorDate: Sat Aug 17 15:03:08 2024 +0000
🔄 synced local 'docs/specification/' with remote 'docs/specification/'
---
docs/specification/java_serialization_spec.md | 72 ++++++++--------
docs/specification/row_format_spec.md | 1 +
docs/specification/xlang_serialization_spec.md | 112 ++++++++++++-------------
3 files changed, 91 insertions(+), 94 deletions(-)
diff --git a/docs/specification/java_serialization_spec.md b/docs/specification/java_serialization_spec.md
index 5bb6511..592413a 100644
--- a/docs/specification/java_serialization_spec.md
+++ b/docs/specification/java_serialization_spec.md
@@ -1,9 +1,11 @@
---
-title: Fury Java Serialization Specification
+title: Fury Java Serialization Format
sidebar_position: 1
id: fury_java_serialization_spec
---
+# Fury Java Serialization Specification
+
## Spec overview
Fury Java Serialization is an automatic object serialization framework that supports reference and polymorphism. Fury
@@ -75,14 +77,14 @@ If schema consistent mode is enabled globally or enabled for current class, clas
- If class is registered, it will be written as a fury unsigned varint: `class_id << 1`.
- If class is not registered:
- - If class is not an array, fury will write one byte `0bxxxxxxx1` first, then write class name.
- - The first little bit is `1`, which is different from first bit `0` of
+ - If class is not an array, fury will write one byte `0bxxxxxxx1` first, then write class name.
+ - The first little bit is `1`, which is different from first bit `0` of
encoded class id. Fury can use this information to determine whether to read class by class id for
deserialization.
- - If class is not registered and class is an array, fury will write one byte `dimensions << 1 | 1` first, then write
+ - If class is not registered and class is an array, fury will write one byte `dimensions << 1 | 1` first, then write
component
class subsequently. This can reduce array class name cost if component class is or will be serialized.
- - Class will be written as two enumerated fury unsigned by default: `package name` and `class name`. If meta share
+ - Class will be written as two enumerated fury unsigned by default: `package name` and `class name`. If meta share
mode is
enabled,
class will be written as an unsigned varint which points to index in `MetaContext`.
@@ -143,10 +145,10 @@ Meta header is a 64 bits number value encoded in little endian order.
```
- num fields: encode `num fields << 1 | register flag(1 when class registered)` as unsigned varint.
- - If class is registered, then an unsigned varint class id will be written next, package and class name will be
+ - If class is registered, then an unsigned varint class id will be written next, package and class name will be
omitted.
- - If current class is schema consistent, then num field will be `0` to flag it.
- - If current class isn't schema consistent, then num field will be the number of compatible fields. For example,
+ - If current class is schema consistent, then num field will be `0` to flag it.
+ - If current class isn't schema consistent, then num field will be the number of compatible fields. For example,
users
can use tag id to mark some field as compatible field in schema consistent context. In such cases, schema
consistent
@@ -154,34 +156,34 @@ Meta header is a 64 bits number value encoded in little endian order.
fields info of those fields which aren't annotated by tag id for deserializing schema consistent fields, then use
fields info in meta for deserializing compatible fields.
- Package name encoding(omitted when class is registered):
- - encoding algorithm: `UTF8/ALL_TO_LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL`
- - Header: `6 bits size | 2 bits encoding flags`. The `6 bits size: 0~63` will be used to indicate size `0~62`,
+ - encoding algorithm: `UTF8/ALL_TO_LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL`
+ - Header: `6 bits size | 2 bits encoding flags`. The `6 bits size: 0~63` will be used to indicate size `0~62`,
the value `63` the size need more byte to read, the encoding will encode `size - 62` as a varint next.
- Class name encoding(omitted when class is registered):
- - encoding algorithm: `UTF8/LOWER_UPPER_DIGIT_SPECIAL/FIRST_TO_LOWER_SPECIAL/ALL_TO_LOWER_SPECIAL`
- - header: `6 bits size | 2 bits encoding flags`. The `6 bits size: 0~63` will be used to indicate size `1~64`,
+ - encoding algorithm: `UTF8/LOWER_UPPER_DIGIT_SPECIAL/FIRST_TO_LOWER_SPECIAL/ALL_TO_LOWER_SPECIAL`
+ - header: `6 bits size | 2 bits encoding flags`. The `6 bits size: 0~63` will be used to indicate size `1~64`,
the value `63` the size need more byte to read, the encoding will encode `size - 63` as a varint next.
- Field info:
- - header(8
+ - header(8
bits): `3 bits size + 2 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag`.
Users can use annotation to provide those info.
- - 2 bits field name encoding:
- - encoding: `UTF8/ALL_TO_LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL/TAG_ID`
- - If tag id is used, i.e. field name is written by an unsigned varint tag id. 2 bits encoding will be `11`.
- - size of field name:
- - The `3 bits size: 0~7` will be used to indicate length `1~7`, the value `6` the size read more bytes,
+ - 2 bits field name encoding:
+ - encoding: `UTF8/ALL_TO_LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL/TAG_ID`
+ - If tag id is used, i.e. field name is written by an unsigned varint tag id. 2 bits encoding will be `11`.
+ - size of field name:
+ - The `3 bits size: 0~7` will be used to indicate length `1~7`, the value `6` the size read more bytes,
the encoding will encode `size - 7` as a varint next.
- - If encoding is `TAG_ID`, then num_bytes of field name will be used to store tag id.
- - ref tracking: when set to 1, ref tracking will be enabled for this field.
- - nullability: when set to 1, this field can be null.
- - polymorphism: when set to 1, the actual type of field will be the declared field type even the type if
+ - If encoding is `TAG_ID`, then num_bytes of field name will be used to store tag id.
+ - ref tracking: when set to 1, ref tracking will be enabled for this field.
+ - nullability: when set to 1, this field can be null.
+ - polymorphism: when set to 1, the actual type of field will be the declared field type even the type if
not `final`.
- - type id:
- - For registered type-consistent classes, it will be the registered class id.
- - Otherwise it will be encoded as `OBJECT_ID` if it isn't `final` and `FINAL_OBJECT_ID` if it's `final`. The
+ - type id:
+ - For registered type-consistent classes, it will be the registered class id.
+ - Otherwise it will be encoded as `OBJECT_ID` if it isn't `final` and `FINAL_OBJECT_ID` if it's `final`. The
meta for such types is written separately instead of inlining here is to reduce meta space cost if object of
this type is serialized in current object graph multiple times, and the field value may be null too.
- - Field name: If type id is set, type id will be used instead. Otherwise meta string encoding length and data will
+ - Field name: If type id is set, type id will be used instead. Otherwise meta string encoding length and data will
be written instead.
Field order are left as implementation details, which is not exposed to specification, the deserialization need to
@@ -193,12 +195,12 @@ using a more compact encoding.
Same encoding algorithm as the previous layer except:
- header + package name:
- - Header:
- - If package name has been written before: `varint index + sharing flag(set)` will be written
- - If package name hasn't been written before:
- - If meta string encoding is `LOWER_SPECIAL` and the length of encoded string `<=` 64, then header will be
+ - Header:
+ - If package name has been written before: `varint index + sharing flag(set)` will be written
+ - If package name hasn't been written before:
+ - If meta string encoding is `LOWER_SPECIAL` and the length of encoded string `<=` 64, then header will be
`6 bits size + encoding flag(set) + sharing flag(unset)`.
- - Otherwise, header will
+ - Otherwise, header will
be `3 bits unset + 3 bits encoding flags + encoding flag(unset) + sharing flag(unset)`
## Meta String
@@ -305,17 +307,17 @@ If string has been written before, the data will be written as follows:
- size: 1~9 byte
- Fury PVL(Progressive Variable-length Long) Encoding:
- - positive long format: first bit in every byte indicates whether to have the next byte. If first bit is set
+ - positive long format: first bit in every byte indicates whether to have the next byte. If first bit is set
i.e. `b & 0x80 == 0x80`, then the next byte should be read until the first bit is unset.
#### Signed long
- size: 1~9 byte
- Fury SLI(Small long as int) Encoding:
- - If long is in [-1073741824, 1073741823], encode as 4 bytes int: `| little-endian: ((int) value) << 1 |`
- - Otherwise write as 9 bytes: `| 0b1 | little-endian 8 bytes long |`
+ - If long is in [-1073741824, 1073741823], encode as 4 bytes int: `| little-endian: ((int) value) << 1 |`
+ - Otherwise write as 9 bytes: `| 0b1 | little-endian 8 bytes long |`
- Fury PVL(Progressive Variable-length Long) Encoding:
- - First convert the number into positive unsigned long by `(v << 1) ^ (v >> 63)` ZigZag algorithm to reduce cost of
+ - First convert the number into positive unsigned long by ` (v << 1) ^ (v >> 63)` ZigZag algorithm to reduce cost of
small negative numbers, then encoding it as an unsigned long.
#### Float
diff --git a/docs/specification/row_format_spec.md b/docs/specification/row_format_spec.md
index af24879..d32c11a 100644
--- a/docs/specification/row_format_spec.md
+++ b/docs/specification/row_format_spec.md
@@ -4,4 +4,5 @@ sidebar_position: 2
id: fury_row_format_spec
---
+# Row Format
Coming soon
diff --git a/docs/specification/xlang_serialization_spec.md b/docs/specification/xlang_serialization_spec.md
index 5f39a08..5882d00 100644
--- a/docs/specification/xlang_serialization_spec.md
+++ b/docs/specification/xlang_serialization_spec.md
@@ -1,11 +1,12 @@
---
-title: Cross-language Serialization Specification
+title: Fury Xlang Serialization Format
sidebar_position: 0
id: fury_xlang_serialization_spec
---
+# Cross-language Serialization Specification
+
> Format Version History:
->
> - Version 0.1 - serialization spec formalized
Fury xlang serialization is an automatic object serialization framework that supports reference and polymorphism.
@@ -41,22 +42,22 @@ also introduce more complexities compared to static serialization frameworks. So
- set: an unordered set of unique elements.
- map: a map of key-value pairs. Mutable types such as `list/map/set/array/tensor/arrow` are not allowed as key of map.
- time types:
- - duration: an absolute length of time, independent of any calendar/timezone, as a count of nanoseconds.
- - timestamp: a point in time, independent of any calendar/timezone, as a count of nanoseconds. The count is relative
+ - duration: an absolute length of time, independent of any calendar/timezone, as a count of nanoseconds.
+ - timestamp: a point in time, independent of any calendar/timezone, as a count of nanoseconds. The count is relative
to an epoch at UTC midnight on January 1, 1970.
- decimal: exact decimal value represented as an integer value in two's complement.
- binary: an variable-length array of bytes.
- array type: only allow numeric components. Other arrays will be taken as List. The implementation should support the
interoperability between array and list.
- - array: multidimensional array which every sub-array can have different sizes but all have same type.
- - bool_array: one dimensional int16 array.
- - int8_array: one dimensional int8 array.
- - int16_array: one dimensional int16 array.
- - int32_array: one dimensional int32 array.
- - int64_array: one dimensional int64 array.
- - float16_array: one dimensional half_float_16 array.
- - float32_array: one dimensional float32 array.
- - float64_array: one dimensional float64 array.
+ - array: multidimensional array which every sub-array can have different sizes but all have same type.
+ - bool_array: one dimensional int16 array.
+ - int8_array: one dimensional int8 array.
+ - int16_array: one dimensional int16 array.
+ - int32_array: one dimensional int32 array.
+ - int64_array: one dimensional int64 array.
+ - float16_array: one dimensional half_float_16 array.
+ - float32_array: one dimensional float32 array.
+ - float64_array: one dimensional float64 array.
- tensor: a multidimensional dense array of fixed-size values such as a NumPy ndarray.
- sparse tensor: a multidimensional array whose elements are almost all zeros.
- arrow record batch: an arrow [record batch](https://arrow.apache.org/docs/cpp/tables.html#record-batches) object.
@@ -196,9 +197,9 @@ differently.
of `type_id`. Schema evolution related meta will be ignored.
- If schema evolution mode is enabled globally when creating fury, and current class is configured to use schema
consistent mode like `struct` vs `table` in flatbuffers:
- - Type meta will be add to `captured_type_defs`: `captured_type_defs[type def stub] = map size` ahead when
+ - Type meta will be add to `captured_type_defs`: `captured_type_defs[type def stub] = map size` ahead when
registering type.
- - Get index of the meta in `captured_type_defs`, write that index as `| unsigned varint: index |`.
+ - Get index of the meta in `captured_type_defs`, write that index as `| unsigned varint: index |`.
### Schema evolution
@@ -206,24 +207,21 @@ If schema evolution mode is enabled globally when creating fury, and enabled for
using one of the following mode. Which mode to use is configured when creating fury.
- Normal mode(meta share not enabled):
- - If type meta hasn't been written before, add `type def`
+ - If type meta hasn't been written before, add `type def`
to `captured_type_defs`: `captured_type_defs[type def] = map size`.
- - Get index of the meta in `captured_type_defs`, write that index as `| unsigned varint: index |`.
- - After finished the serialization of the object graph, fury will start to write `captured_type_defs`:
- - Firstly, set current to `meta start offset` of fury header
- - Then write `captured_type_defs` one by one:
-
- ```python
- buffer.write_var_uint32(len(writting_type_defs) - len(schema_consistent_type_def_stubs))
- for type_meta in writting_type_defs:
- if not type_meta.is_stub():
- type_meta.write_type_def(buffer)
- writing_type_defs = copy(schema_consistent_type_def_stubs)
- ```
-
+ - Get index of the meta in `captured_type_defs`, write that index as `| unsigned varint: index |`.
+ - After finished the serialization of the object graph, fury will start to write `captured_type_defs`:
+ - Firstly, set current to `meta start offset` of fury header
+ - Then write `captured_type_defs` one by one:
+ ```python
+ buffer.write_var_uint32(len(writting_type_defs) - len(schema_consistent_type_def_stubs))
+ for type_meta in writting_type_defs:
+ if not type_meta.is_stub():
+ type_meta.write_type_def(buffer)
+ writing_type_defs = copy(schema_consistent_type_def_stubs)
+ ```
- Meta share mode: the writing steps are same as the normal mode, but `captured_type_defs` will be shared across
multiple serializations of different objects. For example, suppose we have a batch to serialize:
-
```python
captured_type_defs = {}
stream = ...
@@ -236,20 +234,16 @@ using one of the following mode. Which mode to use is configured when creating f
```
- Streaming mode(streaming mode doesn't support meta share):
- - If type meta hasn't been written before, the data will be written as:
-
+ - If type meta hasn't been written before, the data will be written as:
```
| unsigned varint: 0b11111111 | type def |
```
-
- - If type meta has been written before, the data will be written as:
-
+ - If type meta has been written before, the data will be written as:
```
| unsigned varint: written index << 1 |
```
-
`written index` is the id in `captured_type_defs`.
- - With this mode, `meta start offset` can be omitted.
+ - With this mode, `meta start offset` can be omitted.
> The normal mode and meta share mode will forbid streaming writing since it needs to look back for update the start
> offset after the whole object graph writing and meta collecting is finished. Only in this way we can ensure
@@ -287,33 +281,33 @@ Meta header is a 64 bits number value encoded in little endian order.
```
- num fields: encode `num fields` as unsigned varint.
- - If the current type is schema consistent, then num_fields will be `0` to flag it.
- - If the current type isn't schema consistent, then num_fields will be the number of compatible fields. For example,
+ - If the current type is schema consistent, then num_fields will be `0` to flag it.
+ - If the current type isn't schema consistent, then num_fields will be the number of compatible fields. For example,
users can use tag id to mark some fields as compatible fields in schema consistent context. In such cases, schema
consistent fields will be serialized first, then compatible fields will be serialized next. At deserialization,
Fury will use fields info of those fields which aren't annotated by tag id for deserializing schema consistent
fields, then use fields info in meta for deserializing compatible fields.
- type id: the registered id for the current type, which will be written as an unsigned varint.
- field info:
- - header(8
+ - header(8
bits): `3 bits size + 2 bits field name encoding + polymorphism flag + nullability flag + ref tracking flag`.
Users can use annotation to provide those info.
- - 2 bits field name encoding:
- - encoding: `UTF8/ALL_TO_LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL/TAG_ID`
- - If tag id is used, i.e. field name is written by an unsigned varint tag id. 2 bits encoding will be `11`.
- - size of field name:
- - The `3 bits size: 0~7` will be used to indicate length `1~7`, the value `7` indicates to read more bytes,
+ - 2 bits field name encoding:
+ - encoding: `UTF8/ALL_TO_LOWER_SPECIAL/LOWER_UPPER_DIGIT_SPECIAL/TAG_ID`
+ - If tag id is used, i.e. field name is written by an unsigned varint tag id. 2 bits encoding will be `11`.
+ - size of field name:
+ - The `3 bits size: 0~7` will be used to indicate length `1~7`, the value `7` indicates to read more bytes,
the encoding will encode `size - 7` as a varint next.
- - If encoding is `TAG_ID`, then num_bytes of field name will be used to store tag id.
- - ref tracking: when set to 1, ref tracking will be enabled for this field.
- - nullability: when set to 1, this field can be null.
- - polymorphism: when set to 1, the actual type of field will be the declared field type even the type if
+ - If encoding is `TAG_ID`, then num_bytes of field name will be used to store tag id.
+ - ref tracking: when set to 1, ref tracking will be enabled for this field.
+ - nullability: when set to 1, this field can be null.
+ - polymorphism: when set to 1, the actual type of field will be the declared field type even the type if
not `final`.
- - field name: If tag id is set, tag id will be used instead. Otherwise meta string encoding `[length]` and data will
+ - field name: If tag id is set, tag id will be used instead. Otherwise meta string encoding `[length]` and data will
be written instead.
- - type id:
- - For registered type-consistent classes, it will be the registered type id.
- - Otherwise it will be encoded as `OBJECT_ID` if it isn't `final` and `FINAL_OBJECT_ID` if it's `final`. The
+ - type id:
+ - For registered type-consistent classes, it will be the registered type id.
+ - Otherwise it will be encoded as `OBJECT_ID` if it isn't `final` and `FINAL_OBJECT_ID` if it's `final`. The
meta for such types is written separately instead of inlining here is to reduce meta space cost if object of
this type is serialized in current object graph multiple times, and the field value may be null too.
@@ -407,10 +401,10 @@ Notes:
- size: 1~9 byte
- Fury SLI(Small long as int) Encoding:
- - If long is in `[0, 2147483647]`, encode as 4 bytes int: `| little-endian: ((int) value) << 1 |`
- - Otherwise write as 9 bytes: `| 0b1 | little-endian 8 bytes long |`
+ - If long is in `[0, 2147483647]`, encode as 4 bytes int: `| little-endian: ((int) value) << 1 |`
+ - Otherwise write as 9 bytes: `| 0b1 | little-endian 8 bytes long |`
- Fury PVL(Progressive Variable-length Long) Encoding:
- - positive long format: first bit in every byte indicates whether to have the next byte. If first bit is set
+ - positive long format: first bit in every byte indicates whether to have the next byte. If first bit is set
i.e. `b & 0x80 == 0x80`, then the next byte should be read until the first bit is unset.
#### signed int64
@@ -422,10 +416,10 @@ Notes:
- size: 1~9 byte
- Fury SLI(Small long as int) Encoding:
- - If long is in `[-1073741824, 1073741823]`, encode as 4 bytes int: `| little-endian: ((int) value) << 1 |`
- - Otherwise write as 9 bytes: `| 0b1 | little-endian 8 bytes long |`
+ - If long is in `[-1073741824, 1073741823]`, encode as 4 bytes int: `| little-endian: ((int) value) << 1 |`
+ - Otherwise write as 9 bytes: `| 0b1 | little-endian 8 bytes long |`
- Fury PVL(Progressive Variable-length Long) Encoding:
- - First convert the number into positive unsigned long by `(v << 1) ^ (v >> 63)` ZigZag algorithm to reduce cost of
+ - First convert the number into positive unsigned long by `(v << 1) ^ (v >> 63)` ZigZag algorithm to reduce cost of
small negative numbers, then encoding it as an unsigned long.
#### float32
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org