You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fory.apache.org by "mengnankkkk (via GitHub)" <gi...@apache.org> on 2026/02/03 11:17:51 UTC
[PR] feat(java): add float support [fory]
mengnankkkk opened a new pull request, #3254:
URL: https://github.com/apache/fory/pull/3254
<!--
**Thanks for contributing to Apache Fory™.**
**If this is your first time opening a PR on fory, you can refer to [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md).**
Contribution Checklist
- The **Apache Fory™** community has requirements on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md).
- Apache Fory™ has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here.
-->
## Why?
<!-- Describe the purpose of this PR. -->
## What does this PR do?
<!-- Describe the details of this PR. -->
## Related issues
<!--
Is there any related issue? If this PR closes them you say say fix/closes:
- #xxxx0
- #xxxx1
- Fixes #xxxx2
-->
## Does this PR introduce any user-facing change?
<!--
If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary.
Delete section if not applicable.
-->
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
<!--
When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here.
Delete section if not applicable.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [PR] feat(java): add float support [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #3254:
URL: https://github.com/apache/fory/pull/3254#discussion_r2763992582
##########
java/fory-core/src/main/java/org/apache/fory/type/Float16.java:
##########
@@ -0,0 +1,338 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fory.type;
+
+import java.io.Serializable;
+
+public final class Float16 extends Number implements Comparable<Float16>, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private static final int SIGN_MASK = 0x8000;
+ private static final int EXP_MASK = 0x7C00;
+ private static final int MANT_MASK = 0x03FF;
+
+ private static final short BITS_NAN = (short) 0x7E00;
+ private static final short BITS_POS_INF = (short) 0x7C00; // +Inf
+ private static final short BITS_NEG_INF = (short) 0xFC00; // -Inf
+ private static final short BITS_NEG_ZERO = (short) 0x8000; // -0
+ private static final short BITS_MAX = (short) 0x7BFF; // 65504
+ private static final short BITS_ONE = (short) 0x3C00; // 1.0
+ private static final short BITS_MIN_NORMAL = (short) 0x0400;
+ private static final short BITS_MIN_VALUE = (short) 0x0001;
+
+ public static final Float16 NaN = new Float16(BITS_NAN);
+
+ public static final Float16 POSITIVE_INFINITY = new Float16(BITS_POS_INF);
+
+ public static final Float16 NEGATIVE_INFINITY = new Float16(BITS_NEG_INF);
+
+ public static final Float16 ZERO = new Float16((short) 0);
+
+ public static final Float16 NEGATIVE_ZERO = new Float16(BITS_NEG_ZERO);
+
+ public static final Float16 ONE = new Float16(BITS_ONE);
+
+ public static final Float16 MAX_VALUE = new Float16(BITS_MAX);
+
+ public static final Float16 MIN_NORMAL = new Float16(BITS_MIN_NORMAL);
+
+ public static final Float16 MIN_VALUE = new Float16(BITS_MIN_VALUE);
+
+ public static final int SIZE_BITS = 16;
+
+ public static final int SIZE_BYTES = 2;
+
+ private final short bits;
+
+ private Float16(short bits) {
+ this.bits = bits;
+ }
+
+ public static Float16 fromBits(short bits) {
+ return new Float16(bits);
+ }
+
+ public static Float16 valueOf(float value) {
+ return new Float16(floatToFloat16Bits(value));
+ }
+
+ public short toBits() {
+ return bits;
+ }
+
+ public float toFloat() {
+ return floatValue();
+ }
+
+ private static short floatToFloat16Bits(float f32) {
+ int bits32 = Float.floatToRawIntBits(f32);
+ int sign = (bits32 >>> 31) & 0x1;
+ int exp = (bits32 >>> 23) & 0xFF;
+ int mant = bits32 & 0x7FFFFF;
+
+ int outSign = sign << 15;
+ int outExp;
+ int outMant;
+
+ if (exp == 0xFF) {
+ outExp = 0x1F;
+ if (mant != 0) {
+ outMant = 0x200 | ((mant >>> 13) & 0x1FF);
+ if (outMant == 0x200) {
+ outMant = 0x201;
+ }
+ } else {
+ outMant = 0;
+ }
+ } else if (exp == 0) {
+ outExp = 0;
+ outMant = 0;
+ } else {
+ int newExp = exp - 127 + 15;
+
+ if (newExp >= 31) {
+ outExp = 0x1F;
+ outMant = 0;
+ } else if (newExp <= 0) {
+ int fullMant = mant | 0x800000;
+ int shift = 1 - newExp;
+ int netShift = 13 + shift;
+
+ if (netShift >= 24) {
+ outExp = 0;
+ outMant = 0;
+ } else {
+ outExp = 0;
+ int roundBit = (fullMant >>> (netShift - 1)) & 1;
+ int sticky = fullMant & ((1 << (netShift - 1)) - 1);
+ outMant = fullMant >>> netShift;
+
+ if (roundBit == 1 && (sticky != 0 || (outMant & 1) == 1)) {
+ outMant++;
+ }
+ }
+ } else {
+ outExp = newExp;
+ outMant = mant >>> 13;
+
+ int roundBit = (mant >>> 12) & 1;
+ int sticky = mant & 0xFFF;
+
+ if (roundBit == 1 && (sticky != 0 || (outMant & 1) == 1)) {
+ outMant++;
+ if (outMant > 0x3FF) {
+ outMant = 0;
+ outExp++;
+ if (outExp >= 31) {
+ outExp = 0x1F;
+ }
+ }
+ }
+ }
+ }
+
+ return (short) (outSign | (outExp << 10) | outMant);
+ }
+
+ private static float float16BitsToFloat(short bits16) {
+ int bits = bits16 & 0xFFFF;
+ int sign = (bits >>> 15) & 0x1;
+ int exp = (bits >>> 10) & 0x1F;
+ int mant = bits & 0x3FF;
+
+ int outBits = sign << 31;
+
+ if (exp == 0x1F) {
+ outBits |= 0xFF << 23;
+ if (mant != 0) {
+ outBits |= mant << 13;
+ }
+ } else if (exp == 0) {
+ if (mant != 0) {
+ int shift = Integer.numberOfLeadingZeros(mant) - 22;
+ mant = (mant << shift) & 0x3FF;
+ int newExp = 1 - 15 - shift + 127;
+ outBits |= newExp << 23;
+ outBits |= mant << 13;
+ }
+ } else {
+ outBits |= (exp - 15 + 127) << 23;
+ outBits |= mant << 13;
+ }
+
+ return Float.intBitsToFloat(outBits);
+ }
+
+ public boolean isNaN() {
+ return (bits & EXP_MASK) == EXP_MASK && (bits & MANT_MASK) != 0;
+ }
+
+ public boolean isInfinite() {
+ return (bits & EXP_MASK) == EXP_MASK && (bits & MANT_MASK) == 0;
+ }
+
+ public boolean isFinite() {
+ return (bits & EXP_MASK) != EXP_MASK;
+ }
+
+ public boolean isZero() {
+ return (bits & (EXP_MASK | MANT_MASK)) == 0;
+ }
+
+ public boolean isNormal() {
+ int exp = bits & EXP_MASK;
+ return exp != 0 && exp != EXP_MASK;
+ }
+
+ public boolean isSubnormal() {
+ return (bits & EXP_MASK) == 0 && (bits & MANT_MASK) != 0;
+ }
+
+ public boolean signbit() {
+ return (bits & SIGN_MASK) != 0;
+ }
+
+ public Float16 add(Float16 other) {
+ return valueOf(floatValue() + other.floatValue());
+ }
+
+ public Float16 subtract(Float16 other) {
+ return valueOf(floatValue() - other.floatValue());
+ }
+
+ public Float16 multiply(Float16 other) {
+ return valueOf(floatValue() * other.floatValue());
+ }
+
+ public Float16 divide(Float16 other) {
+ return valueOf(floatValue() / other.floatValue());
+ }
+
+ public Float16 negate() {
+ return fromBits((short) (bits ^ SIGN_MASK));
+ }
+
+ public Float16 abs() {
+ return fromBits((short) (bits & ~SIGN_MASK));
+ }
+
+ @Override
+ public float floatValue() {
+ return float16BitsToFloat(bits);
+ }
+
+ @Override
+ public double doubleValue() {
+ return floatValue();
+ }
+
+ @Override
+ public int intValue() {
+ return (int) floatValue();
+ }
+
+ @Override
+ public long longValue() {
+ return (long) floatValue();
+ }
+
+ @Override
+ public byte byteValue() {
+ return (byte) floatValue();
+ }
+
+ @Override
+ public short shortValue() {
+ return (short) floatValue();
+ }
+
+ public boolean equalsValue(Float16 other) {
Review Comment:
```suggestion
public boolean isNumericEqual(Float16 other) {
```
And could you add some java docs for public methods of this class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [PR] feat(java): add float support [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #3254:
URL: https://github.com/apache/fory/pull/3254#discussion_r2758580241
##########
java/fory-core/src/main/java/org/apache/fory/serializer/ArraySerializers.java:
##########
@@ -881,6 +881,52 @@ private void readFloat64BySwapEndian(MemoryBuffer buffer, double[] values, int n
}
}
+ public static final class Float16ArraySerializer
+ extends PrimitiveArraySerializer<org.apache.fory.type.Float16[]> {
Review Comment:
Could you also implement `Float16List` just like we did for `Uint16List`?
And use `short[]` are storage in `Float16List`
##########
java/fory-core/src/main/java/org/apache/fory/serializer/ArraySerializers.java:
##########
@@ -881,6 +881,52 @@ private void readFloat64BySwapEndian(MemoryBuffer buffer, double[] values, int n
}
}
+ public static final class Float16ArraySerializer
+ extends PrimitiveArraySerializer<org.apache.fory.type.Float16[]> {
Review Comment:
Could you also implement `Float16List` just like we did for `Uint16List`?
And use `short[]` as storage in `Float16List`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [PR] feat(java): add float support [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #3254:
URL: https://github.com/apache/fory/pull/3254#discussion_r2868639169
##########
java/fory-core/src/main/java/org/apache/fory/serializer/Float16Serializer.java:
##########
@@ -39,12 +39,10 @@ public Float16 read(MemoryBuffer buffer) {
return Float16.fromBits(buffer.readInt16());
}
- @Override
public void xwrite(MemoryBuffer buffer, Float16 value) {
Review Comment:
remove it, we don' have xwrite/xread methods any more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [PR] feat(java): add float support [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #3254:
URL: https://github.com/apache/fory/pull/3254#discussion_r2802061005
##########
java/fory-core/src/main/java/org/apache/fory/type/DispatchId.java:
##########
@@ -44,20 +44,21 @@ public class DispatchId {
public static final int TAGGED_INT64 = 9;
public static final int FLOAT32 = 10;
public static final int FLOAT64 = 11;
- public static final int UINT8 = 12;
- public static final int UINT16 = 13;
- public static final int UINT32 = 14;
- public static final int VAR_UINT32 = 15;
- public static final int UINT64 = 16;
- public static final int VAR_UINT64 = 17;
- public static final int TAGGED_UINT64 = 18;
- public static final int EXT_UINT8 = 19;
- public static final int EXT_UINT16 = 20;
- public static final int EXT_UINT32 = 21;
- public static final int EXT_VAR_UINT32 = 22;
- public static final int EXT_UINT64 = 23;
- public static final int EXT_VAR_UINT64 = 24;
- public static final int STRING = 25;
+ public static final int FLOAT16 = 12;
Review Comment:
This ID is wrong, here is current id def in main branch:
```
/** float8: an 8-bit floating point number. */
public static final int FLOAT8 = 16;
/** float16: a 16-bit floating point number. */
public static final int FLOAT16 = 17;
/** bfloat16: a 16-bit brain floating point number. */
public static final int BFLOAT16 = 18;
```
Same for Float16 array
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [PR] feat(java): add float support [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #3254:
URL: https://github.com/apache/fory/pull/3254#discussion_r2758580241
##########
java/fory-core/src/main/java/org/apache/fory/serializer/ArraySerializers.java:
##########
@@ -881,6 +881,52 @@ private void readFloat64BySwapEndian(MemoryBuffer buffer, double[] values, int n
}
}
+ public static final class Float16ArraySerializer
+ extends PrimitiveArraySerializer<org.apache.fory.type.Float16[]> {
Review Comment:
Could you also implement `Float16List` under `org.apache.fory.collection` package just like we did for `Uint16List`?
And use `short[]` as storage in `Float16List`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [PR] feat(java): add float support [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #3254:
URL: https://github.com/apache/fory/pull/3254#discussion_r2836930543
##########
java/fory-core/src/main/java/org/apache/fory/type/DispatchId.java:
##########
@@ -49,15 +49,16 @@ public class DispatchId {
public static final int UINT32 = 14;
public static final int VAR_UINT32 = 15;
public static final int UINT64 = 16;
- public static final int VAR_UINT64 = 17;
- public static final int TAGGED_UINT64 = 18;
- public static final int EXT_UINT8 = 19;
- public static final int EXT_UINT16 = 20;
- public static final int EXT_UINT32 = 21;
- public static final int EXT_VAR_UINT32 = 22;
- public static final int EXT_UINT64 = 23;
- public static final int EXT_VAR_UINT64 = 24;
- public static final int STRING = 25;
+ public static final int FLOAT16 = 17;
+ public static final int VAR_UINT64 = 18;
+ public static final int TAGGED_UINT64 = 19;
+ public static final int EXT_UINT8 = 20;
+ public static final int EXT_UINT16 = 21;
+ public static final int EXT_UINT32 = 22;
+ public static final int EXT_VAR_UINT32 = 23;
+ public static final int EXT_UINT64 = 24;
+ public static final int EXT_VAR_UINT64 = 25;
+ public static final int STRING = 26;
Review Comment:
java/fory-core/src/main/java/org/apache/fory/serializer/FieldGroups.java also needs update to clarify Float16 as build-in fields and grouped into primitive/boxed groups based whethther it's nullable.
java/fory-core/src/main/java/org/apache/fory/builder/BaseObjectCodecBuilder.java also needs update to handle Flaot16 specially, you can start from `serializePrimitiveField`
java/fory-core/src/main/java/org/apache/fory/serializer/AbstractObjectSerializer.java may also needs updates
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [PR] feat(java): add float support [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #3254:
URL: https://github.com/apache/fory/pull/3254#discussion_r2764010507
##########
java/fory-core/src/main/resources/META-INF/native-image/org.apache.fory/fory-core/reflection-config.json:
##########
@@ -5,5 +5,24 @@
"allPublicConstructors": true,
"allDeclaredMethods": true,
"allPublicMethods": true
+ },
+ {
+ "name": "org.apache.fory.type.Float16",
+ "allDeclaredConstructors": true,
+ "allPublicConstructors": true,
+ "allDeclaredMethods": true,
+ "allPublicMethods": true,
+ "allDeclaredFields": true,
+ "allPublicFields": true
+ },
+ {
+ "name": "org.apache.fory.collection.Float16List",
Review Comment:
Could you share the reason that you think it's necessary to register `Float16List/Float16`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [PR] feat(java): add float support [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #3254:
URL: https://github.com/apache/fory/pull/3254#discussion_r2764005876
##########
java/fory-core/src/main/resources/META-INF/native-image/org.apache.fory/fory-core/serialization-config.json:
##########
@@ -16,5 +16,6 @@
{ "name": "java.lang.NullPointerException" },
{ "name": "java.lang.IndexOutOfBoundsException" },
{ "name": "java.lang.ArrayIndexOutOfBoundsException" },
- { "name": "java.io.IOException" }
+ { "name": "java.io.IOException" },
+ { "name": "org.apache.fory.type.Float16" }
Review Comment:
this is also unnecessary. Float16Serializer should be registerd in fory as an internal serializer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [PR] feat(java): add float support [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #3254:
URL: https://github.com/apache/fory/pull/3254#discussion_r2764002452
##########
java/fory-core/src/main/resources/META-INF/native-image/org.apache.fory/fory-core/reflection-config.json:
##########
@@ -5,5 +5,24 @@
"allPublicConstructors": true,
"allDeclaredMethods": true,
"allPublicMethods": true
+ },
+ {
+ "name": "org.apache.fory.type.Float16",
+ "allDeclaredConstructors": true,
+ "allPublicConstructors": true,
+ "allDeclaredMethods": true,
+ "allPublicMethods": true,
+ "allDeclaredFields": true,
Review Comment:
do we need to expose fields?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [PR] feat(java): add float support [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #3254:
URL: https://github.com/apache/fory/pull/3254#discussion_r2890818920
##########
java/fory-core/src/main/java/org/apache/fory/serializer/ArraySerializers.java:
##########
@@ -868,6 +869,146 @@ private void readFloat64BySwapEndian(MemoryBuffer buffer, double[] values, int n
}
}
+ public static final class Float16ArraySerializer extends PrimitiveArraySerializer<Float16[]> {
+ private static final int NULLABLE_ENCODING_FLAG = 1;
+
+ public Float16ArraySerializer(Fory fory) {
+ super(fory, Float16[].class);
+ }
+
+ @Override
+ public void write(MemoryBuffer buffer, Float16[] value) {
+ int length = value.length;
+ boolean hasNull = false;
+ for (int i = 0; i < length; i++) {
+ if (value[i] == null) {
+ hasNull = true;
+ break;
+ }
+ }
+ if (hasNull) {
+ writeNullable(buffer, value, length);
+ return;
+ }
+ writeNonNull(buffer, value, length);
+ }
+
+ private void writeNonNull(MemoryBuffer buffer, Float16[] value, int length) {
+ int size = length * 2;
+ buffer.writeVarUint32Small7(size);
+
+ if (Platform.IS_LITTLE_ENDIAN) {
+ int writerIndex = buffer.writerIndex();
+ buffer.ensure(writerIndex + size);
+ for (int i = 0; i < length; i++) {
+ buffer._unsafePutInt16(writerIndex + i * 2, value[i].toBits());
+ }
+ buffer._unsafeWriterIndex(writerIndex + size);
+ } else {
+ for (int i = 0; i < length; i++) {
+ buffer.writeInt16(value[i].toBits());
+ }
+ }
+ }
+
+ private void writeNullable(MemoryBuffer buffer, Float16[] value, int length) {
+ int bitmapSize = (length + 7) >>> 3;
+ int payloadSize = Integer.BYTES + bitmapSize + length * 2;
+ int encodedSize = (payloadSize << 1) | NULLABLE_ENCODING_FLAG;
Review Comment:
This changes the FLOAT16_ARRAY wire layout when nulls are present, which will break cross-language readers that expect the normal length+payload format for type 53.
In xlang mode, other runtimes decode FLOAT16_ARRAY as plain [byte_length][2-byte elements]. Here we switch to an alternate encoded-size scheme for nullable arrays, so non-Java runtimes will misread the payload. Could we keep FLOAT16_ARRAY encoding stable and handle nullable cases another way (for example, fallback to LIST encoding when null is present)?
##########
java/fory-core/src/main/java/org/apache/fory/type/DispatchId.java:
##########
@@ -49,19 +49,23 @@ public class DispatchId {
public static final int UINT32 = 14;
public static final int VAR_UINT32 = 15;
public static final int UINT64 = 16;
- public static final int VAR_UINT64 = 17;
- public static final int TAGGED_UINT64 = 18;
- public static final int EXT_UINT8 = 19;
- public static final int EXT_UINT16 = 20;
- public static final int EXT_UINT32 = 21;
- public static final int EXT_VAR_UINT32 = 22;
- public static final int EXT_UINT64 = 23;
- public static final int EXT_VAR_UINT64 = 24;
- public static final int STRING = 25;
+ public static final int FLOAT16 = 17;
Review Comment:
Since FLOAT16 is now a basic dispatch id, FieldSkipper also needs a skip branch for it.
Right now FieldSkipper has no DispatchId.FLOAT16 case. In compatibility reads, skipping an unknown Float16 field will throw Unexpected basic dispatchId instead of advancing by 2 bytes. Please add FLOAT16 to the 2-byte skip branch.
##########
java/fory-core/src/main/java/org/apache/fory/serializer/Serializers.java:
##########
@@ -570,6 +571,7 @@ public static void registerDefaultSerializers(Fory fory) {
resolver.registerInternalSerializer(URI.class, new URISerializer(fory));
resolver.registerInternalSerializer(Pattern.class, new RegexSerializer(fory));
resolver.registerInternalSerializer(UUID.class, new UUIDSerializer(fory));
+ resolver.registerInternalSerializer(Float16.class, new Float16Serializer(fory));
Review Comment:
Float16 gets registered twice (primitive serializer path and general serializers path).
This overrides the earlier registration and makes it harder to reason about which serializer is intended. It would be cleaner to keep a single registration point for Float16.
##########
compiler/fory_compiler/generators/java.py:
##########
@@ -1458,7 +1460,11 @@ def generate_equals_method(self, message: Message) -> List[str]:
)
elif isinstance(field.field_type, PrimitiveType):
kind = field.field_type.kind
- if kind in (PrimitiveKind.FLOAT32,):
+ if kind in (PrimitiveKind.FLOAT16,):
+ comparisons.append(
+ f"({field_name} == null ? that.{field_name} == null : (that.{field_name} != null && {field_name}.equalsValue(that.{field_name})))"
Review Comment:
Generated equals/hashCode for float16 now use numeric semantics, but Float16.equals/hashCode are bitwise.
With this change, +0 and -0 compare equal in generated messages, and NaN is not equal to NaN, while Float16 itself uses raw-bit equality. That mismatch is surprising in user code. Can we align generated message behavior with Float16.equals/hashCode (or make the rule explicit and consistent across runtimes)?
##########
java/fory-core/src/main/java/org/apache/fory/collection/Float16List.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fory.collection;
+
+import java.util.AbstractList;
+import java.util.Arrays;
+import java.util.Objects;
+import java.util.RandomAccess;
+import org.apache.fory.type.Float16;
+
+public final class Float16List extends AbstractList<Float16> implements RandomAccess {
+ private static final int DEFAULT_CAPACITY = 10;
+
+ private short[] array;
+ private int size;
+
+ public Float16List() {
+ this(DEFAULT_CAPACITY);
+ }
+
+ public Float16List(int initialCapacity) {
+ if (initialCapacity < 0) {
+ throw new IllegalArgumentException("Illegal capacity: " + initialCapacity);
+ }
+ this.array = new short[initialCapacity];
+ this.size = 0;
+ }
+
+ public Float16List(short[] array) {
+ this.array = array;
+ this.size = array.length;
+ }
+
+ @Override
+ public Float16 get(int index) {
+ checkIndex(index);
+ return Float16.fromBits(array[index]);
+ }
+
+ @Override
+ public int size() {
+ return size;
+ }
+
+ @Override
+ public Float16 set(int index, Float16 element) {
+ checkIndex(index);
+ Objects.requireNonNull(element, "element");
+ short prev = array[index];
+ array[index] = element.toBits();
+ return Float16.fromBits(prev);
+ }
+
+ public void set(int index, short bits) {
+ checkIndex(index);
+ array[index] = bits;
+ }
+
+ public void set(int index, float value) {
+ checkIndex(index);
+ array[index] = Float16.valueOf(value).toBits();
+ }
+
+ @Override
+ public void add(int index, Float16 element) {
+ checkPositionIndex(index);
+ ensureCapacity(size + 1);
+ System.arraycopy(array, index, array, index + 1, size - index);
+ array[index] = element.toBits();
+ size++;
+ modCount++;
+ }
+
+ @Override
+ public boolean add(Float16 element) {
+ Objects.requireNonNull(element, "element");
+ ensureCapacity(size + 1);
+ array[size++] = element.toBits();
+ modCount++;
+ return true;
+ }
+
+ public boolean add(short bits) {
+ ensureCapacity(size + 1);
+ array[size++] = bits;
+ modCount++;
+ return true;
+ }
+
+ public boolean add(float value) {
Review Comment:
Float16List primitive methods still allocate Float16 objects on the hot path.
add(float), set(float), and getFloat(int) all go through Float16.valueOf/fromBits, so these paths allocate even though this is a primitive-style container. A static bits conversion helper would keep these methods allocation-free.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [PR] feat(java): add float support [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #3254:
URL: https://github.com/apache/fory/pull/3254#discussion_r2836922331
##########
java/fory-core/src/main/resources/META-INF/native-image/org.apache.fory/fory-core/reflection-config.json:
##########
@@ -5,5 +5,8 @@
"allPublicConstructors": true,
"allDeclaredMethods": true,
"allPublicMethods": true
+ },
Review Comment:
This is syntan error, please fix it
##########
java/fory-core/src/main/resources/META-INF/native-image/org.apache.fory/fory-core/reflection-config.json:
##########
@@ -5,5 +5,8 @@
"allPublicConstructors": true,
"allDeclaredMethods": true,
"allPublicMethods": true
+ },
Review Comment:
This is syntax error, please fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [PR] feat(java): add float support [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #3254:
URL: https://github.com/apache/fory/pull/3254#discussion_r2836925097
##########
java/fory-core/src/test/java/org/apache/fory/serializer/Float16SerializerTest.java:
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fory.serializer;
+
+import static org.testng.Assert.*;
+
+import lombok.AllArgsConstructor;
+import lombok.Data;
+import org.apache.fory.Fory;
+import org.apache.fory.ForyTestBase;
+import org.apache.fory.config.Language;
+import org.apache.fory.type.Float16;
+import org.testng.annotations.Test;
+
+public class Float16SerializerTest extends ForyTestBase {
+
+ @Test
+ public void testFloat16Serialization() {
+ Fory fory = Fory.builder().withLanguage(Language.JAVA).requireClassRegistration(false).build();
+
+ byte[] bytes = fory.serialize(Float16.NaN);
+ Float16 result = (Float16) fory.deserialize(bytes);
+ assertTrue(result.isNaN());
+
+ bytes = fory.serialize(Float16.POSITIVE_INFINITY);
+ result = (Float16) fory.deserialize(bytes);
+ assertTrue(result.isInfinite() && !result.signbit());
+
+ bytes = fory.serialize(Float16.NEGATIVE_INFINITY);
+ result = (Float16) fory.deserialize(bytes);
+ assertTrue(result.isInfinite() && result.signbit());
+
+ bytes = fory.serialize(Float16.ZERO);
+ result = (Float16) fory.deserialize(bytes);
+ assertEquals(Float16.ZERO.toBits(), result.toBits());
+
+ bytes = fory.serialize(Float16.ONE);
+ result = (Float16) fory.deserialize(bytes);
+ assertEquals(Float16.ONE.toBits(), result.toBits());
+
+ bytes = fory.serialize(Float16.valueOf(1.5f));
+ result = (Float16) fory.deserialize(bytes);
+ assertEquals(1.5f, result.floatValue(), 0.01f);
+ }
+
+ @Test
+ public void testFloat16ArraySerialization() {
+ Fory fory = Fory.builder().withLanguage(Language.JAVA).requireClassRegistration(false).build();
+
+ Float16[] array =
+ new Float16[] {
+ Float16.ZERO,
+ Float16.ONE,
+ Float16.valueOf(2.5f),
+ Float16.valueOf(-1.5f),
+ Float16.NaN,
+ Float16.POSITIVE_INFINITY,
+ Float16.NEGATIVE_INFINITY,
+ Float16.MAX_VALUE,
+ Float16.MIN_VALUE
+ };
+
+ byte[] bytes = fory.serialize(array);
+ Float16[] result = (Float16[]) fory.deserialize(bytes);
+
+ assertEquals(array.length, result.length);
+ for (int i = 0; i < array.length; i++) {
+ if (array[i].isNaN()) {
+ assertTrue(result[i].isNaN(), "Index " + i + " should be NaN");
+ } else {
+ assertEquals(
+ array[i].toBits(),
+ result[i].toBits(),
+ "Index "
+ + i
+ + " bits should match: expected 0x"
+ + Integer.toHexString(array[i].toBits() & 0xFFFF)
+ + " but got 0x"
+ + Integer.toHexString(result[i].toBits() & 0xFFFF));
+ }
+ }
+ }
+
+ @Test
+ public void testFloat16EmptyArray() {
+ Fory fory = Fory.builder().withLanguage(Language.JAVA).requireClassRegistration(false).build();
+
+ Float16[] empty = new Float16[0];
+ byte[] bytes = fory.serialize(empty);
+ Float16[] result = (Float16[]) fory.deserialize(bytes);
+ assertEquals(0, result.length);
+ }
+
+ @Test
+ public void testFloat16LargeArray() {
+ Fory fory = Fory.builder().withLanguage(Language.JAVA).requireClassRegistration(false).build();
+
+ Float16[] large = new Float16[1000];
+ for (int i = 0; i < large.length; i++) {
+ large[i] = Float16.valueOf((float) i);
+ }
+
+ byte[] bytes = fory.serialize(large);
+ Float16[] result = (Float16[]) fory.deserialize(bytes);
+
+ assertEquals(large.length, result.length);
+ for (int i = 0; i < large.length; i++) {
+ assertEquals(
+ large[i].floatValue(), result[i].floatValue(), 0.1f, "Index " + i + " should match");
+ }
+ }
+
+ @Data
+ @AllArgsConstructor
+ public static class StructWithFloat16 {
+ Float16 f16Field;
+ Float16 f16Field2;
+ }
+
+ @Test
+ public void testStructWithFloat16Field() {
+ Fory fory = Fory.builder().withLanguage(Language.JAVA).requireClassRegistration(false).build();
+ fory.register(StructWithFloat16.class);
+
+ StructWithFloat16 obj = new StructWithFloat16(Float16.valueOf(1.5f), Float16.valueOf(-2.5f));
+
+ byte[] bytes = fory.serialize(obj);
+ StructWithFloat16 result = (StructWithFloat16) fory.deserialize(bytes);
+
+ assertEquals(obj.f16Field.toBits(), result.f16Field.toBits());
+ assertEquals(obj.f16Field2.toBits(), result.f16Field2.toBits());
+ }
+
+ @Data
+ @AllArgsConstructor
+ public static class StructWithFloat16Array {
+ Float16[] f16Array;
+ }
+
+ @Test
+ public void testStructWithFloat16Array() {
+ Fory fory = Fory.builder().withLanguage(Language.JAVA).requireClassRegistration(false).build();
+ fory.register(StructWithFloat16Array.class);
+
+ Float16[] array = new Float16[] {Float16.ONE, Float16.valueOf(2.0f), Float16.valueOf(3.0f)};
+ StructWithFloat16Array obj = new StructWithFloat16Array(array);
+
+ byte[] bytes = fory.serialize(obj);
+ StructWithFloat16Array result = (StructWithFloat16Array) fory.deserialize(bytes);
+
+ assertEquals(obj.f16Array.length, result.f16Array.length);
+ for (int i = 0; i < obj.f16Array.length; i++) {
+ assertEquals(obj.f16Array[i].toBits(), result.f16Array[i].toBits());
+ }
+ }
+
+ @Test
+ public void testFloat16WithNullableField() {
+ Fory fory = Fory.builder().withLanguage(Language.JAVA).requireClassRegistration(false).build();
+
+ @Data
+ @AllArgsConstructor
+ class StructWithNullableFloat16 {
+ Float16 nullableField;
+ }
+
+ fory.register(StructWithNullableFloat16.class);
+
+ StructWithNullableFloat16 obj1 = new StructWithNullableFloat16(Float16.valueOf(1.5f));
+ byte[] bytes = fory.serialize(obj1);
+ StructWithNullableFloat16 result = (StructWithNullableFloat16) fory.deserialize(bytes);
+ assertEquals(obj1.nullableField.toBits(), result.nullableField.toBits());
+
+ StructWithNullableFloat16 obj2 = new StructWithNullableFloat16(null);
+ bytes = fory.serialize(obj2);
+ result = (StructWithNullableFloat16) fory.deserialize(bytes);
+ assertNull(result.nullableField);
+ }
+
+ @Test
+ public void testFloat16SpecialValuesInStruct() {
+ Fory fory = Fory.builder().withLanguage(Language.JAVA).requireClassRegistration(false).build();
+
+ @Data
+ @AllArgsConstructor
+ class StructWithSpecialValues {
Review Comment:
Please also create another tests to struct with Float16List as fields and also test Float16List as top level objects for serialization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [PR] feat(java): add float support [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang merged PR #3254:
URL: https://github.com/apache/fory/pull/3254
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org