You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fury.apache.org by "HuangXingBo (via GitHub)" <gi...@apache.org> on 2024/10/18 06:31:27 UTC
[PR] [WIP] string optimize poc [fury]
HuangXingBo opened a new pull request, #1890:
URL: https://github.com/apache/fury/pull/1890
<!--
**Thanks for contributing to Fury.**
**If this is your first time opening a PR on fury, you can refer to [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md).**
Contribution Checklist
- The **Apache Fury (incubating)** community has restrictions on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md).
- Fury has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here.
-->
## What does this PR do?
<!-- Describe the purpose of this PR. -->
## Related issues
<!--
Is there any related issue? Please attach here.
- #xxxx0
- #xxxx1
- #xxxx2
-->
## Does this PR introduce any user-facing change?
<!--
If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary.
-->
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
<!--
When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): Refactor String serialization and deserialization [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang merged PR #1890:
URL: https://github.com/apache/fury/pull/1890
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): Refactor String serialization and deserialization [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1808630319
##########
java/fury-core/src/main/java/org/apache/fury/serializer/StringSerializer.java:
##########
@@ -603,26 +722,121 @@ private static MethodHandle getJavaStringZeroCopyCtrHandle() {
}
}
- public void writeUTF8String(MemoryBuffer buffer, String value) {
- byte[] bytes = value.getBytes(StandardCharsets.UTF_8);
- buffer.writeVarUint32(bytes.length);
- buffer.writeBytes(bytes);
+ private static void heapWriteCharsUTF16BE(
+ char[] chars, int arrIndex, int numBytes, byte[] targetArray) {
+ // Write to heap memory then copy is 250% faster than unsafe write to direct memory.
+ int charIndex = 0;
+ for (int i = arrIndex, end = i + numBytes; i < end; i += 2) {
+ char c = chars[charIndex++];
+ targetArray[i] = (byte) (c >> StringUTF16.HI_BYTE_SHIFT);
+ targetArray[i + 1] = (byte) (c >> StringUTF16.LO_BYTE_SHIFT);
+ }
}
- public String readUTF8String(MemoryBuffer buffer) {
- int numBytes = buffer.readVarUint32Small14();
- buffer.checkReadableBytes(numBytes);
- final byte[] targetArray = buffer.getHeapMemory();
- if (targetArray != null) {
- String str =
- new String(
- targetArray, buffer._unsafeHeapReaderIndex(), numBytes, StandardCharsets.UTF_8);
- buffer.increaseReaderIndex(numBytes);
- return str;
+ private int offHeapWriteCharsUTF16(
+ MemoryBuffer buffer, char[] chars, int writerIndex, int numBytes) {
+ byte[] tmpArray = getByteArray(numBytes);
+ int charIndex = 0;
+ for (int i = 0; i < numBytes; i += 2) {
+ char c = chars[charIndex++];
+ tmpArray[i] = (byte) (c >> StringUTF16.HI_BYTE_SHIFT);
+ tmpArray[i + 1] = (byte) (c >> StringUTF16.LO_BYTE_SHIFT);
+ }
+ buffer.put(writerIndex, tmpArray, 0, numBytes);
+ writerIndex += numBytes;
+ return writerIndex;
+ }
+
+ private static byte bestCoder(char[] chars) {
+ int numChars = chars.length;
+ // sample 64 chars
+ int sampleNum = Math.min(64, numChars);
+ int vectorizedLen = sampleNum >> 2;
+ int vectorizedChars = vectorizedLen << 2;
+ int endOffset = Platform.CHAR_ARRAY_OFFSET + (vectorizedChars << 1);
+ int count = 0;
+ for (int offset = Platform.CHAR_ARRAY_OFFSET, charOffset = 0;
+ offset < endOffset;
+ offset += 8, charOffset += 4) {
+ long multiChars = Platform.getLong(chars, offset);
+ if ((multiChars & MULTI_CHARS_NON_LATIN_MASK) == 0) {
+ count += 4;
+ } else {
+ for (int i = 0; i < 4; ++i) {
+ if (chars[charOffset + i] < 0x80) {
+ count++;
+ }
+ }
+ }
+ }
+
+ for (int i = vectorizedChars; i < sampleNum; i++) {
+ if (chars[i] < 0x80) {
+ count++;
+ }
+ }
+
+ // ascii number > 50%, choose UTF-8
+ if (count >= sampleNum * 0.5) {
+ if (count == sampleNum && StringUtils.isLatin(chars)) {
Review Comment:
Could we use `StringUtils.isLatin(chars, sampleNum)` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): Refactor String serialization and deserialization [fury]
Posted by "HuangXingBo (via GitHub)" <gi...@apache.org>.
HuangXingBo commented on PR #1890:
URL: https://github.com/apache/fury/pull/1890#issuecomment-2426890397
my private GHA has passed. https://github.com/HuangXingBo/fury/actions/runs/11442511611
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] [WIP] string optimize poc [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1805960612
##########
java/fury-core/src/main/java/org/apache/fury/serializer/StringSerializer.java:
##########
@@ -625,4 +743,534 @@ public String readUTF8String(MemoryBuffer buffer) {
return new String(tmpArray, 0, numBytes, StandardCharsets.UTF_8);
}
}
+
+ public byte[] readBytesLatin1(MemoryBuffer buffer) {
+ int utf8AsciiBytes = buffer.readInt32();
+ buffer.checkReadableBytes(utf8AsciiBytes);
+ byte[] srcArray = buffer.getHeapMemory();
+ byte[] bytes = new byte[utf8AsciiBytes << 1];
+ if (srcArray != null) {
+ int srcIndex = buffer._unsafeHeapReaderIndex();
+ for (int i = 0, pos = Platform.IS_LITTLE_ENDIAN ? 0 : 1; i < utf8AsciiBytes; ++i, pos += 2) {
+ bytes[pos] = srcArray[srcIndex++];
+ }
+ buffer._increaseReaderIndexUnsafe(utf8AsciiBytes);
+ } else {
+ // TODO: off-heap
+ }
+ return bytes;
+ }
+
+ public char[] readCharsLatin1(MemoryBuffer buffer) {
+ int utf8AsciiBytes = buffer.readInt32();
+ buffer.checkReadableBytes(utf8AsciiBytes);
+ byte[] srcArray = buffer.getHeapMemory();
+ char[] chars = new char[utf8AsciiBytes];
+ if (srcArray != null) {
+ int srcIndex = buffer._unsafeHeapReaderIndex();
+ for (int i = 0; i < utf8AsciiBytes; i++) {
+ chars[i] = (char) (srcArray[srcIndex++] & 0xff);
+ }
+ buffer._increaseReaderIndexUnsafe(utf8AsciiBytes);
+ } else {
+ // TODO: off-heap
+ }
+ return chars;
+ }
+
+ public byte[] readBytesUTF8(MemoryBuffer buffer) {
+ int utf16Bytes = buffer.readInt32();
+ int udf8Bytes = buffer.readInt32();
+ byte[] bytes = new byte[utf16Bytes];
+ buffer.checkReadableBytes(udf8Bytes);
+ byte[] srcArray = buffer.getHeapMemory();
+ if (srcArray != null) {
+ int srcIndex = buffer._unsafeHeapReaderIndex();
+ if (!fastDecodeUTF8(srcArray, srcIndex, udf8Bytes, bytes)) {
+ throw new RuntimeException("Decode failed");
+ }
+ buffer._increaseReaderIndexUnsafe(udf8Bytes);
+ } else {
+ // TODO: off-heap
+ }
+ return bytes;
+ }
+
+ public byte[] readBytesUTF16(MemoryBuffer buffer) {
+ int utf16Bytes = buffer.readInt32();
+ buffer.checkReadableBytes(utf16Bytes);
+ byte[] bytes;
+ byte[] heapMemory = buffer.getHeapMemory();
+ if (heapMemory != null) {
+ final int arrIndex = buffer._unsafeHeapReaderIndex();
+ buffer.increaseReaderIndex(utf16Bytes);
+ bytes = new byte[utf16Bytes];
+ System.arraycopy(heapMemory, arrIndex, bytes, 0, utf16Bytes);
+ } else {
+ bytes = buffer.readBytes(utf16Bytes);
+ }
+ return bytes;
+ }
+
+ public char[] readCharsUTF16(MemoryBuffer buffer) {
+ int utf16Bytes = buffer.readInt32();
+ char[] chars = new char[utf16Bytes >> 1];
+ if (Platform.IS_LITTLE_ENDIAN) {
+ // FIXME JDK11 utf16 string uses little-endian order.
+ buffer.readChars(chars, Platform.CHAR_ARRAY_OFFSET, utf16Bytes);
+ } else {
+ buffer.checkReadableBytes(utf16Bytes);
+ final byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int charIndex = 0;
+ for (int i = buffer._unsafeHeapReaderIndex(), end = i + utf16Bytes; i < end; i += 2) {
+ char c =
+ (char)
+ ((targetArray[i] & 0xff << StringUTF16.HI_BYTE_SHIFT)
+ | ((targetArray[i + 1] & 0xff) << StringUTF16.LO_BYTE_SHIFT));
+ chars[charIndex++] = c;
+ }
+ buffer._increaseReaderIndexUnsafe(utf16Bytes);
+ } else {
+ final byte[] tmpArray = getByteArray(utf16Bytes);
+ buffer.readBytes(tmpArray, 0, utf16Bytes);
+ int charIndex = 0;
+ for (int i = 0; i < utf16Bytes; i += 2) {
+ char c =
+ (char)
+ ((tmpArray[i] & 0xff << StringUTF16.HI_BYTE_SHIFT)
+ | ((tmpArray[i + 1] & 0xff) << StringUTF16.LO_BYTE_SHIFT));
+ chars[charIndex++] = c;
+ }
+ }
+ }
+ return chars;
+ }
+
+ public char[] readCharsUTF8(MemoryBuffer buffer) {
+ int utf16Bytes = buffer.readInt32();
+ int udf8Bytes = buffer.readInt32();
+ char[] chars = new char[utf16Bytes >> 1];
+ buffer.checkReadableBytes(udf8Bytes);
+ byte[] srcArray = buffer.getHeapMemory();
+ if (srcArray != null) {
+ int srcIndex = buffer._unsafeHeapReaderIndex();
+ if (!fastDecodeUTF8(srcArray, srcIndex, udf8Bytes, chars)) {
+ throw new RuntimeException("Decode failed");
+ }
+ buffer._increaseReaderIndexUnsafe(udf8Bytes);
+ } else {
+ // TODO: off-heap
+ }
+ return chars;
+ }
+
+ public void writeBytesLatin1(MemoryBuffer buffer, byte[] bytes) {
+ int writerIndex = buffer.writerIndex();
+ int numBytes = bytes.length >> 1;
+ buffer.ensure(writerIndex + 5 + numBytes);
+ byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int arrIndex = buffer._unsafeHeapWriterIndex();
+ buffer.putByte(arrIndex, LATIN1);
+ buffer.putInt32(arrIndex + 1, numBytes);
+ arrIndex += 5;
+ for (int i = Platform.IS_LITTLE_ENDIAN ? 0 : 1; i < numBytes; i += 2) {
+ targetArray[arrIndex++] = bytes[i];
+ }
+ writerIndex += 5;
+ } else {
+ // TODO: off-heap
+ }
+ writerIndex += numBytes;
+ buffer._unsafeWriterIndex(writerIndex);
+ }
+
+ public void writeCharsLatin1(MemoryBuffer buffer, char[] chars) {
+ int writerIndex = buffer.writerIndex();
+ int numBytes = chars.length;
+ buffer.ensure(writerIndex + 5 + numBytes);
+ byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int arrIndex = buffer._unsafeHeapWriterIndex();
+ buffer.putByte(arrIndex, LATIN1);
+ buffer.putInt32(arrIndex + 1, numBytes);
+ arrIndex += 5;
+ for (int i = 0; i < numBytes; i++) {
+ targetArray[arrIndex + i] = (byte) chars[i];
+ }
+ writerIndex += 5;
+ } else {
+ // TODO: off-heap
+ }
+ writerIndex += numBytes;
+ buffer._unsafeWriterIndex(writerIndex);
+ }
+
+ public void writeBytesUTF16(MemoryBuffer buffer, byte[] bytes) {
+ int numBytes = bytes.length;
+ int writerIndex = buffer.writerIndex();
+ buffer.ensure(writerIndex + 5 + numBytes);
+ final byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int arrIndex = buffer._unsafeHeapWriterIndex();
+ buffer.putByte(arrIndex, UTF16);
+ buffer.putInt32(arrIndex + 1, numBytes);
+ arrIndex += 5;
+ writerIndex += 5 + numBytes;
+ System.arraycopy(bytes, 0, targetArray, arrIndex, numBytes);
+ } else {
+ // TODO: off-heap
+ }
+ buffer._unsafeWriterIndex(writerIndex);
+ }
+
+ public void writeCharsUTF16(MemoryBuffer buffer, char[] chars) {
+ int numBytes = MathUtils.doubleExact(chars.length);
+ int writerIndex = buffer.writerIndex();
+ buffer.ensure(writerIndex + 5 + numBytes);
+ final byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int arrIndex = buffer._unsafeHeapWriterIndex();
+ buffer.putByte(arrIndex, UTF16);
+ buffer.putInt32(arrIndex + 1, numBytes);
+ arrIndex += 5;
+ writerIndex += 5 + numBytes;
+ if (Platform.IS_LITTLE_ENDIAN) {
+ // FIXME JDK11 utf16 string uses little-endian order.
+ Platform.UNSAFE.copyMemory(
+ chars,
+ Platform.CHAR_ARRAY_OFFSET,
+ targetArray,
+ Platform.BYTE_ARRAY_OFFSET + arrIndex,
+ numBytes);
+ } else {
+ heapWriteCharsUTF16BE(chars, arrIndex, numBytes, targetArray);
+ }
+ } else {
+ // TODO: off-heap
+ }
+ buffer._unsafeWriterIndex(writerIndex);
+ }
+
+ public void writeCharsUTF8(MemoryBuffer buffer, char[] chars) {
+ int estimateMaxBytes = chars.length * 3;
+ int writerIndex = buffer.writerIndex();
+ buffer.ensure(writerIndex + 9 + estimateMaxBytes);
+ byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int arrIndex = buffer._unsafeHeapWriterIndex();
+ int targetIndex = fastEncodeUTF8(chars, targetArray, arrIndex + 9);
+ int written = targetIndex - arrIndex - 9;
+ buffer.putByte(arrIndex, UTF8);
+ buffer.putInt32(arrIndex + 1, chars.length << 1);
+ buffer.putInt32(arrIndex + 5, written);
+ buffer._unsafeWriterIndex(targetIndex);
+ } else {
+ // TODO: off-heap
+ }
+ }
+
+ public void writeBytesUTF8(MemoryBuffer buffer, byte[] bytes) {
+ int estimateMaxBytes = bytes.length / 2 * 3;
+ int writerIndex = buffer.writerIndex();
+ buffer.ensure(writerIndex + 9 + estimateMaxBytes);
+ byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int arrIndex = buffer._unsafeHeapWriterIndex();
+ int targetIndex = fastEncodeUTF8(bytes, targetArray, arrIndex + 9);
+ int written = targetIndex - arrIndex - 9;
+ buffer.putByte(arrIndex, UTF8);
+ buffer.putInt32(arrIndex + 1, bytes.length);
+ buffer.putInt32(arrIndex + 5, written);
+ buffer._unsafeWriterIndex(targetIndex);
+ }
+ }
+
+ private static boolean fastDecodeUTF8(byte[] src, int offset, int len, byte[] dst) {
+ final int end = offset + len;
+ int dp = 0;
+
+ while (offset < end) {
+ if (offset + 8 <= end
+ && (Platform.getLong(src, Platform.BYTE_ARRAY_OFFSET + offset) & 0x8080808080808080L)
+ == 0) {
+ // ascii only
+ for (int i = 0, pos = Platform.IS_LITTLE_ENDIAN ? dp : dp + 1; i < 8; ++i, pos += 2) {
+ dst[pos] = src[offset++];
+ }
+ dp += 16;
+ } else {
+ int b0 = src[offset++];
+ if (b0 >= 0) {
+ // 1 byte, 7 bits: 0xxxxxxx
+ dst[dp] = (byte) b0;
+ dst[dp + 1] = 0;
+ dp += 2;
+ } else if ((b0 >> 5) == -2 && (b0 & 0x1e) != 0) {
+ // 2 bytes, 11 bits: 110xxxxx 10xxxxxx
+ if (offset >= end) {
+ return false;
+ }
+ int b1 = src[offset++];
+ if ((b1 & 0xc0) != 0x80) { // isNotContinuation(b2)
+ return false;
+ } else {
+ char c = (char) (((b0 << 6) ^ b1) ^ (((byte) 0xC0 << 6) ^ ((byte) 0x80)));
+ dst[dp] = (byte) c;
+ dst[dp + 1] = (byte) (c >> 8);
+ dp += 2;
+ }
+ } else if ((b0 >> 4) == -2) {
+ // 3 bytes, 16 bits: 1110xxxx 10xxxxxx 10xxxxxx
+ if (offset + 1 >= end) {
+ return false;
+ }
+ int b1 = src[offset];
+ int b2 = src[offset + 1];
+ offset += 2;
+ if ((b0 == (byte) 0xe0 && (b1 & 0xe0) == 0x80) //
+ || (b1 & 0xc0) != 0x80 //
+ || (b2 & 0xc0) != 0x80) { // isMalformed3(b0, b1, b2)
+ return false;
+ } else {
+ char c =
+ (char)
+ ((b0 << 12)
+ ^ (b1 << 6)
+ ^ (b2 ^ (((byte) 0xE0 << 12) ^ ((byte) 0x80 << 6) ^ ((byte) 0x80))));
+ boolean isSurrogate = c >= '\uD800' && c < ('\uDFFF' + 1);
+ if (isSurrogate) {
+ return false;
+ } else {
+ dst[dp] = (byte) c;
+ dst[dp + 1] = (byte) (c >> 8);
+ dp += 2;
+ }
+ }
+ } else if ((b0 >> 3) == -2) {
+ // 4 bytes, 21 bits: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
+ if (offset + 2 >= end) {
+ return false;
+ }
+ int b2 = src[offset];
+ int b3 = src[offset + 1];
+ int b4 = src[offset + 2];
+ offset += 3;
+ int uc =
+ ((b0 << 18)
+ ^ (b2 << 12)
+ ^ (b3 << 6)
+ ^ (b4
+ ^ (((byte) 0xF0 << 18)
+ ^ ((byte) 0x80 << 12)
+ ^ ((byte) 0x80 << 6)
+ ^ ((byte) 0x80))));
+ if (((b2 & 0xc0) != 0x80 || (b3 & 0xc0) != 0x80 || (b4 & 0xc0) != 0x80) // isMalformed4
+ ||
+ // shortest form check
+ !(uc >= 0x010000 && uc < 0X10FFFF + 1) // !Character.isSupplementaryCodePoint(uc)
+ ) {
+ return false;
+ } else {
+ char c = (char) ((uc >>> 10) + ('\uD800' - (0x010000 >>> 10)));
+ dst[dp] = (byte) c;
+ dst[dp + 1] = (byte) (c >> 8);
+ dp += 2;
+
+ c = (char) ((uc & 0x3ff) + '\uDC00');
+ dst[dp] = (byte) c;
+ dst[dp + 1] = (byte) (c >> 8);
+ dp += 2;
+ }
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ private static boolean fastDecodeUTF8(byte[] src, int offset, int len, char[] dst) {
+ int end = offset + len;
+ int dp = 0;
+ while (offset < end) {
+ if (offset + 8 <= end
+ && (Platform.getLong(src, Platform.BYTE_ARRAY_OFFSET + offset) & 0x8080808080808080L)
+ == 0) {
+ // ascii only
+ for (int i = 0; i < 8; ++i) {
Review Comment:
ditto for loop unroll
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): Refactor String serialization and deserialization [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1808666752
##########
java/fury-core/src/main/java/org/apache/fury/serializer/StringSerializer.java:
##########
@@ -235,75 +214,106 @@ public String readBytesString(MemoryBuffer buffer) {
}
@CodegenInvoke
- public String readCompressedCharsString(MemoryBuffer buffer) {
+ public String readCharsString(MemoryBuffer buffer) {
+ return newCharsStringZeroCopy(buffer.readCharsAndSize());
+ }
+
+ @CodegenInvoke
+ public String readCompressedBytesString(MemoryBuffer buffer) {
long header = buffer.readVarUint36Small();
byte coder = (byte) (header & 0b11);
int numBytes = (int) (header >>> 2);
- if (coder == LATIN1) {
- return newCharsStringZeroCopy(readLatinChars(buffer, numBytes));
- } else if (coder == UTF16) {
- return newCharsStringZeroCopy(readUTF16Chars(buffer, numBytes));
+ if (coder == UTF8) {
+ return newBytesStringZeroCopy(UTF16, readBytesUTF8(buffer, numBytes));
+ } else if (coder == LATIN1 || coder == UTF16) {
+ return newBytesStringZeroCopy(coder, readBytesUnCompressedUTF16(buffer, numBytes));
} else {
- return readUtf8(buffer, coder, numBytes);
+ throw new RuntimeException("Unknown coder type " + coder);
}
}
- private String readUtf8(MemoryBuffer buffer, byte coder, int numBytes) {
- Preconditions.checkArgument(coder == UTF8, UTF8);
- byte[] bytes = buffer.readBytes(numBytes);
- return new String(bytes, 0, numBytes, StandardCharsets.UTF_8);
- }
-
- private byte[] getByteArray(int numElements) {
- byte[] byteArray = this.byteArray;
- if (byteArray.length < numElements) {
- byteArray = new byte[numElements];
- this.byteArray = byteArray;
- }
- if (byteArray.length > DEFAULT_BUFFER_SIZE) {
- smoothByteArrayLength =
- Math.max(((int) (smoothByteArrayLength * 0.9 + numElements * 0.1)), DEFAULT_BUFFER_SIZE);
- if (smoothByteArrayLength <= DEFAULT_BUFFER_SIZE) {
- this.byteArray = new byte[DEFAULT_BUFFER_SIZE];
- }
+ @CodegenInvoke
+ public String readCompressedCharsString(MemoryBuffer buffer) {
+ long header = buffer.readVarUint36Small();
+ byte coder = (byte) (header & 0b11);
+ int numBytes = (int) (header >>> 2);
+ char[] chars;
+ if (coder == LATIN1) {
+ chars = readCharsLatin1(buffer, numBytes);
+ } else if (coder == UTF8) {
+ chars = readCharsUTF8(buffer, numBytes);
+ } else if (coder == UTF16) {
+ chars = readCharsUTF16(buffer, numBytes);
+ } else {
+ throw new RuntimeException("Unknown coder type " + coder);
}
- return byteArray;
+ return newCharsStringZeroCopy(chars);
}
// Invoked by fury JIT
public void writeJavaString(MemoryBuffer buffer, String value) {
if (STRING_VALUE_FIELD_IS_BYTES) {
- writeBytesString(buffer, value);
+ if (compressString) {
+ writeCompressedBytesString(buffer, value);
+ } else {
+ writeBytesString(buffer, value);
+ }
} else {
assert STRING_VALUE_FIELD_IS_CHARS;
- final char[] chars = (char[]) Platform.getObject(value, STRING_VALUE_FIELD_OFFSET);
if (compressString) {
- if (StringUtils.isLatin(chars)) {
- writeCharsLatin(buffer, chars, chars.length);
- } else {
- writeCharsUTF16(buffer, chars, chars.length);
- }
+ writeCompressedCharsString(buffer, value);
} else {
- int numBytes = MathUtils.doubleExact(value.length());
- buffer.writePrimitiveArrayWithSize(chars, Platform.CHAR_ARRAY_OFFSET, numBytes);
+ writeCharsString(buffer, value);
}
}
}
+ public void writeUTF8String(MemoryBuffer buffer, String value) {
+ byte[] bytes = value.getBytes(StandardCharsets.UTF_8);
+ buffer.writeVarUint32(bytes.length);
+ buffer.writeBytes(bytes);
+ }
+
// Invoked by fury JIT
public String readJavaString(MemoryBuffer buffer) {
if (STRING_VALUE_FIELD_IS_BYTES) {
- return readBytesString(buffer);
+ if (compressString) {
+ return readCompressedBytesString(buffer);
+ } else {
+ return readBytesString(buffer);
+ }
} else {
assert STRING_VALUE_FIELD_IS_CHARS;
if (compressString) {
return readCompressedCharsString(buffer);
} else {
- return newCharsStringZeroCopy(buffer.readCharsAndSize());
+ return readCharsString(buffer);
}
}
}
+ public void writeCompressedBytesString(MemoryBuffer buffer, String value) {
+ final byte[] bytes = (byte[]) Platform.getObject(value, STRING_VALUE_FIELD_OFFSET);
+ final byte coder = Platform.getByte(value, Offset.STRING_CODER_FIELD_OFFSET);
+ if (coder == LATIN1 || bestCoder(bytes) == UTF16) {
+ writeBytesString(buffer, value);
Review Comment:
Maybe we should make `writeBytesString` receive bytes and coder, and invoke like `writeBytesString(buffer, coder, bytes)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): Refactor String serialization and deserialization [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1807807093
##########
java/fury-core/src/main/java/org/apache/fury/util/StringEncodingUtils.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fury.util;
+
+import static org.apache.fury.util.StringUtils.MULTI_CHARS_NON_LATIN_MASK;
+
+import org.apache.fury.memory.Platform;
+
+/** String Encoding Utils. */
+public class StringEncodingUtils {
+
Review Comment:
```suggestion
/**
* A fast encoding implementation to convert a utf16 encoded string into utf8 encoded
* string when most chars in the string are ascii chars.
*/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] [WIP] string optimize poc [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on PR #1890:
URL: https://github.com/apache/fury/pull/1890#issuecomment-2421543583
The code in StringSerializer started to grow a lot, maybe we could move UTF8/16 encoding to a new file `org.apache.fury.serializer.StringUTF8`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] [WIP] string optimize poc [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1805960170
##########
java/fury-core/src/main/java/org/apache/fury/serializer/StringSerializer.java:
##########
@@ -625,4 +743,534 @@ public String readUTF8String(MemoryBuffer buffer) {
return new String(tmpArray, 0, numBytes, StandardCharsets.UTF_8);
}
}
+
+ public byte[] readBytesLatin1(MemoryBuffer buffer) {
+ int utf8AsciiBytes = buffer.readInt32();
+ buffer.checkReadableBytes(utf8AsciiBytes);
+ byte[] srcArray = buffer.getHeapMemory();
+ byte[] bytes = new byte[utf8AsciiBytes << 1];
+ if (srcArray != null) {
+ int srcIndex = buffer._unsafeHeapReaderIndex();
+ for (int i = 0, pos = Platform.IS_LITTLE_ENDIAN ? 0 : 1; i < utf8AsciiBytes; ++i, pos += 2) {
+ bytes[pos] = srcArray[srcIndex++];
+ }
+ buffer._increaseReaderIndexUnsafe(utf8AsciiBytes);
+ } else {
+ // TODO: off-heap
+ }
+ return bytes;
+ }
+
+ public char[] readCharsLatin1(MemoryBuffer buffer) {
+ int utf8AsciiBytes = buffer.readInt32();
+ buffer.checkReadableBytes(utf8AsciiBytes);
+ byte[] srcArray = buffer.getHeapMemory();
+ char[] chars = new char[utf8AsciiBytes];
+ if (srcArray != null) {
+ int srcIndex = buffer._unsafeHeapReaderIndex();
+ for (int i = 0; i < utf8AsciiBytes; i++) {
+ chars[i] = (char) (srcArray[srcIndex++] & 0xff);
+ }
+ buffer._increaseReaderIndexUnsafe(utf8AsciiBytes);
+ } else {
+ // TODO: off-heap
+ }
+ return chars;
+ }
+
+ public byte[] readBytesUTF8(MemoryBuffer buffer) {
+ int utf16Bytes = buffer.readInt32();
+ int udf8Bytes = buffer.readInt32();
+ byte[] bytes = new byte[utf16Bytes];
+ buffer.checkReadableBytes(udf8Bytes);
+ byte[] srcArray = buffer.getHeapMemory();
+ if (srcArray != null) {
+ int srcIndex = buffer._unsafeHeapReaderIndex();
+ if (!fastDecodeUTF8(srcArray, srcIndex, udf8Bytes, bytes)) {
+ throw new RuntimeException("Decode failed");
+ }
+ buffer._increaseReaderIndexUnsafe(udf8Bytes);
+ } else {
+ // TODO: off-heap
+ }
+ return bytes;
+ }
+
+ public byte[] readBytesUTF16(MemoryBuffer buffer) {
+ int utf16Bytes = buffer.readInt32();
+ buffer.checkReadableBytes(utf16Bytes);
+ byte[] bytes;
+ byte[] heapMemory = buffer.getHeapMemory();
+ if (heapMemory != null) {
+ final int arrIndex = buffer._unsafeHeapReaderIndex();
+ buffer.increaseReaderIndex(utf16Bytes);
+ bytes = new byte[utf16Bytes];
+ System.arraycopy(heapMemory, arrIndex, bytes, 0, utf16Bytes);
+ } else {
+ bytes = buffer.readBytes(utf16Bytes);
+ }
+ return bytes;
+ }
+
+ public char[] readCharsUTF16(MemoryBuffer buffer) {
+ int utf16Bytes = buffer.readInt32();
+ char[] chars = new char[utf16Bytes >> 1];
+ if (Platform.IS_LITTLE_ENDIAN) {
+ // FIXME JDK11 utf16 string uses little-endian order.
+ buffer.readChars(chars, Platform.CHAR_ARRAY_OFFSET, utf16Bytes);
+ } else {
+ buffer.checkReadableBytes(utf16Bytes);
+ final byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int charIndex = 0;
+ for (int i = buffer._unsafeHeapReaderIndex(), end = i + utf16Bytes; i < end; i += 2) {
+ char c =
+ (char)
+ ((targetArray[i] & 0xff << StringUTF16.HI_BYTE_SHIFT)
+ | ((targetArray[i + 1] & 0xff) << StringUTF16.LO_BYTE_SHIFT));
+ chars[charIndex++] = c;
+ }
+ buffer._increaseReaderIndexUnsafe(utf16Bytes);
+ } else {
+ final byte[] tmpArray = getByteArray(utf16Bytes);
+ buffer.readBytes(tmpArray, 0, utf16Bytes);
+ int charIndex = 0;
+ for (int i = 0; i < utf16Bytes; i += 2) {
+ char c =
+ (char)
+ ((tmpArray[i] & 0xff << StringUTF16.HI_BYTE_SHIFT)
+ | ((tmpArray[i + 1] & 0xff) << StringUTF16.LO_BYTE_SHIFT));
+ chars[charIndex++] = c;
+ }
+ }
+ }
+ return chars;
+ }
+
+ public char[] readCharsUTF8(MemoryBuffer buffer) {
+ int utf16Bytes = buffer.readInt32();
+ int udf8Bytes = buffer.readInt32();
+ char[] chars = new char[utf16Bytes >> 1];
+ buffer.checkReadableBytes(udf8Bytes);
+ byte[] srcArray = buffer.getHeapMemory();
+ if (srcArray != null) {
+ int srcIndex = buffer._unsafeHeapReaderIndex();
+ if (!fastDecodeUTF8(srcArray, srcIndex, udf8Bytes, chars)) {
+ throw new RuntimeException("Decode failed");
+ }
+ buffer._increaseReaderIndexUnsafe(udf8Bytes);
+ } else {
+ // TODO: off-heap
+ }
+ return chars;
+ }
+
+ public void writeBytesLatin1(MemoryBuffer buffer, byte[] bytes) {
+ int writerIndex = buffer.writerIndex();
+ int numBytes = bytes.length >> 1;
+ buffer.ensure(writerIndex + 5 + numBytes);
+ byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int arrIndex = buffer._unsafeHeapWriterIndex();
+ buffer.putByte(arrIndex, LATIN1);
+ buffer.putInt32(arrIndex + 1, numBytes);
+ arrIndex += 5;
+ for (int i = Platform.IS_LITTLE_ENDIAN ? 0 : 1; i < numBytes; i += 2) {
+ targetArray[arrIndex++] = bytes[i];
+ }
+ writerIndex += 5;
+ } else {
+ // TODO: off-heap
+ }
+ writerIndex += numBytes;
+ buffer._unsafeWriterIndex(writerIndex);
+ }
+
+ public void writeCharsLatin1(MemoryBuffer buffer, char[] chars) {
+ int writerIndex = buffer.writerIndex();
+ int numBytes = chars.length;
+ buffer.ensure(writerIndex + 5 + numBytes);
+ byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int arrIndex = buffer._unsafeHeapWriterIndex();
+ buffer.putByte(arrIndex, LATIN1);
+ buffer.putInt32(arrIndex + 1, numBytes);
+ arrIndex += 5;
+ for (int i = 0; i < numBytes; i++) {
+ targetArray[arrIndex + i] = (byte) chars[i];
+ }
+ writerIndex += 5;
+ } else {
+ // TODO: off-heap
+ }
+ writerIndex += numBytes;
+ buffer._unsafeWriterIndex(writerIndex);
+ }
+
+ public void writeBytesUTF16(MemoryBuffer buffer, byte[] bytes) {
+ int numBytes = bytes.length;
+ int writerIndex = buffer.writerIndex();
+ buffer.ensure(writerIndex + 5 + numBytes);
+ final byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int arrIndex = buffer._unsafeHeapWriterIndex();
+ buffer.putByte(arrIndex, UTF16);
+ buffer.putInt32(arrIndex + 1, numBytes);
+ arrIndex += 5;
+ writerIndex += 5 + numBytes;
+ System.arraycopy(bytes, 0, targetArray, arrIndex, numBytes);
+ } else {
+ // TODO: off-heap
+ }
+ buffer._unsafeWriterIndex(writerIndex);
+ }
+
+ public void writeCharsUTF16(MemoryBuffer buffer, char[] chars) {
+ int numBytes = MathUtils.doubleExact(chars.length);
+ int writerIndex = buffer.writerIndex();
+ buffer.ensure(writerIndex + 5 + numBytes);
+ final byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int arrIndex = buffer._unsafeHeapWriterIndex();
+ buffer.putByte(arrIndex, UTF16);
+ buffer.putInt32(arrIndex + 1, numBytes);
+ arrIndex += 5;
+ writerIndex += 5 + numBytes;
+ if (Platform.IS_LITTLE_ENDIAN) {
+ // FIXME JDK11 utf16 string uses little-endian order.
+ Platform.UNSAFE.copyMemory(
+ chars,
+ Platform.CHAR_ARRAY_OFFSET,
+ targetArray,
+ Platform.BYTE_ARRAY_OFFSET + arrIndex,
+ numBytes);
+ } else {
+ heapWriteCharsUTF16BE(chars, arrIndex, numBytes, targetArray);
+ }
+ } else {
+ // TODO: off-heap
+ }
+ buffer._unsafeWriterIndex(writerIndex);
+ }
+
+ public void writeCharsUTF8(MemoryBuffer buffer, char[] chars) {
+ int estimateMaxBytes = chars.length * 3;
+ int writerIndex = buffer.writerIndex();
+ buffer.ensure(writerIndex + 9 + estimateMaxBytes);
+ byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int arrIndex = buffer._unsafeHeapWriterIndex();
+ int targetIndex = fastEncodeUTF8(chars, targetArray, arrIndex + 9);
+ int written = targetIndex - arrIndex - 9;
+ buffer.putByte(arrIndex, UTF8);
+ buffer.putInt32(arrIndex + 1, chars.length << 1);
+ buffer.putInt32(arrIndex + 5, written);
+ buffer._unsafeWriterIndex(targetIndex);
+ } else {
+ // TODO: off-heap
+ }
+ }
+
+ public void writeBytesUTF8(MemoryBuffer buffer, byte[] bytes) {
+ int estimateMaxBytes = bytes.length / 2 * 3;
+ int writerIndex = buffer.writerIndex();
+ buffer.ensure(writerIndex + 9 + estimateMaxBytes);
+ byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int arrIndex = buffer._unsafeHeapWriterIndex();
+ int targetIndex = fastEncodeUTF8(bytes, targetArray, arrIndex + 9);
+ int written = targetIndex - arrIndex - 9;
+ buffer.putByte(arrIndex, UTF8);
+ buffer.putInt32(arrIndex + 1, bytes.length);
+ buffer.putInt32(arrIndex + 5, written);
+ buffer._unsafeWriterIndex(targetIndex);
+ }
+ }
+
+ private static boolean fastDecodeUTF8(byte[] src, int offset, int len, byte[] dst) {
+ final int end = offset + len;
+ int dp = 0;
+
+ while (offset < end) {
+ if (offset + 8 <= end
+ && (Platform.getLong(src, Platform.BYTE_ARRAY_OFFSET + offset) & 0x8080808080808080L)
+ == 0) {
+ // ascii only
+ for (int i = 0, pos = Platform.IS_LITTLE_ENDIAN ? dp : dp + 1; i < 8; ++i, pos += 2) {
Review Comment:
Could we try to unroll this loop into 8 lines of code and test whether will it be faster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): Refactor String serialization and deserialization [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1807805568
##########
java/fury-core/src/main/java/org/apache/fury/util/StringEncodingUtils.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fury.util;
+
+import static org.apache.fury.util.StringUtils.MULTI_CHARS_NON_LATIN_MASK;
+
+import org.apache.fury.memory.Platform;
+
+/** String Encoding Utils. */
+public class StringEncodingUtils {
+
+ public static int convertUTF16ToUTF8(char[] src, byte[] dst, int dp) {
+ int numChars = src.length;
+ for (int charOffset = 0; charOffset < numChars; ) {
+ if (charOffset + 4 <= numChars
+ && (Platform.getLong(src, Platform.CHAR_ARRAY_OFFSET + charOffset * 2L)
+ & MULTI_CHARS_NON_LATIN_MASK)
+ == 0) {
+ // ascii only
+ dst[dp] = (byte) src[charOffset];
+ dst[dp + 1] = (byte) src[charOffset + 1];
+ dst[dp + 2] = (byte) src[charOffset + 2];
+ dst[dp + 3] = (byte) src[charOffset + 3];
+ dp += 4;
+ charOffset += 4;
+ } else {
+ char c = src[charOffset++];
+ if (c < 0x80) {
+ dst[dp++] = (byte) c;
+ } else if (c < 0x800) {
+ dst[dp] = (byte) (0xc0 | (c >> 6));
+ dst[dp + 1] = (byte) (0x80 | (c & 0x3f));
+ dp += 2;
+ } else if (c >= '\uD800' && c <= Character.MAX_LOW_SURROGATE) {
Review Comment:
Since this method is called only when there are many ascii chars, this branch is less frequent. If this big method can't be inlined, we can move this branch into a new method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): Refactor String serialization and deserialization [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1807807093
##########
java/fury-core/src/main/java/org/apache/fury/util/StringEncodingUtils.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fury.util;
+
+import static org.apache.fury.util.StringUtils.MULTI_CHARS_NON_LATIN_MASK;
+
+import org.apache.fury.memory.Platform;
+
+/** String Encoding Utils. */
+public class StringEncodingUtils {
+
Review Comment:
```suggestion
A fast encoding implementation to convert a utf16 encoded string into utf8 encoded binary when most chars in the string are ascii chars.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] [WIP] string optimize poc [fury]
Posted by "HuangXingBo (via GitHub)" <gi...@apache.org>.
HuangXingBo commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1806618380
##########
java/fury-core/src/main/java/org/apache/fury/serializer/StringSerializer.java:
##########
@@ -625,4 +743,534 @@ public String readUTF8String(MemoryBuffer buffer) {
return new String(tmpArray, 0, numBytes, StandardCharsets.UTF_8);
}
}
+
+ public byte[] readBytesLatin1(MemoryBuffer buffer) {
+ int utf8AsciiBytes = buffer.readInt32();
+ buffer.checkReadableBytes(utf8AsciiBytes);
+ byte[] srcArray = buffer.getHeapMemory();
+ byte[] bytes = new byte[utf8AsciiBytes << 1];
+ if (srcArray != null) {
+ int srcIndex = buffer._unsafeHeapReaderIndex();
+ for (int i = 0, pos = Platform.IS_LITTLE_ENDIAN ? 0 : 1; i < utf8AsciiBytes; ++i, pos += 2) {
+ bytes[pos] = srcArray[srcIndex++];
+ }
+ buffer._increaseReaderIndexUnsafe(utf8AsciiBytes);
+ } else {
+ // TODO: off-heap
+ }
+ return bytes;
+ }
+
+ public char[] readCharsLatin1(MemoryBuffer buffer) {
+ int utf8AsciiBytes = buffer.readInt32();
+ buffer.checkReadableBytes(utf8AsciiBytes);
+ byte[] srcArray = buffer.getHeapMemory();
+ char[] chars = new char[utf8AsciiBytes];
+ if (srcArray != null) {
+ int srcIndex = buffer._unsafeHeapReaderIndex();
+ for (int i = 0; i < utf8AsciiBytes; i++) {
+ chars[i] = (char) (srcArray[srcIndex++] & 0xff);
+ }
+ buffer._increaseReaderIndexUnsafe(utf8AsciiBytes);
+ } else {
+ // TODO: off-heap
+ }
+ return chars;
+ }
+
+ public byte[] readBytesUTF8(MemoryBuffer buffer) {
+ int utf16Bytes = buffer.readInt32();
+ int udf8Bytes = buffer.readInt32();
+ byte[] bytes = new byte[utf16Bytes];
+ buffer.checkReadableBytes(udf8Bytes);
+ byte[] srcArray = buffer.getHeapMemory();
+ if (srcArray != null) {
+ int srcIndex = buffer._unsafeHeapReaderIndex();
+ if (!fastDecodeUTF8(srcArray, srcIndex, udf8Bytes, bytes)) {
+ throw new RuntimeException("Decode failed");
+ }
+ buffer._increaseReaderIndexUnsafe(udf8Bytes);
+ } else {
+ // TODO: off-heap
+ }
+ return bytes;
+ }
+
+ public byte[] readBytesUTF16(MemoryBuffer buffer) {
+ int utf16Bytes = buffer.readInt32();
+ buffer.checkReadableBytes(utf16Bytes);
+ byte[] bytes;
+ byte[] heapMemory = buffer.getHeapMemory();
+ if (heapMemory != null) {
+ final int arrIndex = buffer._unsafeHeapReaderIndex();
+ buffer.increaseReaderIndex(utf16Bytes);
+ bytes = new byte[utf16Bytes];
+ System.arraycopy(heapMemory, arrIndex, bytes, 0, utf16Bytes);
+ } else {
+ bytes = buffer.readBytes(utf16Bytes);
+ }
+ return bytes;
+ }
+
+ public char[] readCharsUTF16(MemoryBuffer buffer) {
+ int utf16Bytes = buffer.readInt32();
+ char[] chars = new char[utf16Bytes >> 1];
+ if (Platform.IS_LITTLE_ENDIAN) {
+ // FIXME JDK11 utf16 string uses little-endian order.
+ buffer.readChars(chars, Platform.CHAR_ARRAY_OFFSET, utf16Bytes);
+ } else {
+ buffer.checkReadableBytes(utf16Bytes);
+ final byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int charIndex = 0;
+ for (int i = buffer._unsafeHeapReaderIndex(), end = i + utf16Bytes; i < end; i += 2) {
+ char c =
+ (char)
+ ((targetArray[i] & 0xff << StringUTF16.HI_BYTE_SHIFT)
+ | ((targetArray[i + 1] & 0xff) << StringUTF16.LO_BYTE_SHIFT));
+ chars[charIndex++] = c;
+ }
+ buffer._increaseReaderIndexUnsafe(utf16Bytes);
+ } else {
+ final byte[] tmpArray = getByteArray(utf16Bytes);
+ buffer.readBytes(tmpArray, 0, utf16Bytes);
+ int charIndex = 0;
+ for (int i = 0; i < utf16Bytes; i += 2) {
+ char c =
+ (char)
+ ((tmpArray[i] & 0xff << StringUTF16.HI_BYTE_SHIFT)
+ | ((tmpArray[i + 1] & 0xff) << StringUTF16.LO_BYTE_SHIFT));
+ chars[charIndex++] = c;
+ }
+ }
+ }
+ return chars;
+ }
+
+ public char[] readCharsUTF8(MemoryBuffer buffer) {
+ int utf16Bytes = buffer.readInt32();
+ int udf8Bytes = buffer.readInt32();
+ char[] chars = new char[utf16Bytes >> 1];
+ buffer.checkReadableBytes(udf8Bytes);
+ byte[] srcArray = buffer.getHeapMemory();
+ if (srcArray != null) {
+ int srcIndex = buffer._unsafeHeapReaderIndex();
+ if (!fastDecodeUTF8(srcArray, srcIndex, udf8Bytes, chars)) {
+ throw new RuntimeException("Decode failed");
+ }
+ buffer._increaseReaderIndexUnsafe(udf8Bytes);
+ } else {
+ // TODO: off-heap
+ }
+ return chars;
+ }
+
+ public void writeBytesLatin1(MemoryBuffer buffer, byte[] bytes) {
+ int writerIndex = buffer.writerIndex();
+ int numBytes = bytes.length >> 1;
+ buffer.ensure(writerIndex + 5 + numBytes);
+ byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int arrIndex = buffer._unsafeHeapWriterIndex();
+ buffer.putByte(arrIndex, LATIN1);
+ buffer.putInt32(arrIndex + 1, numBytes);
+ arrIndex += 5;
+ for (int i = Platform.IS_LITTLE_ENDIAN ? 0 : 1; i < numBytes; i += 2) {
+ targetArray[arrIndex++] = bytes[i];
+ }
+ writerIndex += 5;
+ } else {
+ // TODO: off-heap
+ }
+ writerIndex += numBytes;
+ buffer._unsafeWriterIndex(writerIndex);
+ }
+
+ public void writeCharsLatin1(MemoryBuffer buffer, char[] chars) {
+ int writerIndex = buffer.writerIndex();
+ int numBytes = chars.length;
+ buffer.ensure(writerIndex + 5 + numBytes);
+ byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int arrIndex = buffer._unsafeHeapWriterIndex();
+ buffer.putByte(arrIndex, LATIN1);
+ buffer.putInt32(arrIndex + 1, numBytes);
+ arrIndex += 5;
+ for (int i = 0; i < numBytes; i++) {
+ targetArray[arrIndex + i] = (byte) chars[i];
+ }
+ writerIndex += 5;
+ } else {
+ // TODO: off-heap
+ }
+ writerIndex += numBytes;
+ buffer._unsafeWriterIndex(writerIndex);
+ }
+
+ public void writeBytesUTF16(MemoryBuffer buffer, byte[] bytes) {
+ int numBytes = bytes.length;
+ int writerIndex = buffer.writerIndex();
+ buffer.ensure(writerIndex + 5 + numBytes);
+ final byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int arrIndex = buffer._unsafeHeapWriterIndex();
+ buffer.putByte(arrIndex, UTF16);
+ buffer.putInt32(arrIndex + 1, numBytes);
+ arrIndex += 5;
+ writerIndex += 5 + numBytes;
+ System.arraycopy(bytes, 0, targetArray, arrIndex, numBytes);
+ } else {
+ // TODO: off-heap
+ }
+ buffer._unsafeWriterIndex(writerIndex);
+ }
+
+ public void writeCharsUTF16(MemoryBuffer buffer, char[] chars) {
+ int numBytes = MathUtils.doubleExact(chars.length);
+ int writerIndex = buffer.writerIndex();
+ buffer.ensure(writerIndex + 5 + numBytes);
+ final byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int arrIndex = buffer._unsafeHeapWriterIndex();
+ buffer.putByte(arrIndex, UTF16);
+ buffer.putInt32(arrIndex + 1, numBytes);
+ arrIndex += 5;
+ writerIndex += 5 + numBytes;
+ if (Platform.IS_LITTLE_ENDIAN) {
+ // FIXME JDK11 utf16 string uses little-endian order.
+ Platform.UNSAFE.copyMemory(
+ chars,
+ Platform.CHAR_ARRAY_OFFSET,
+ targetArray,
+ Platform.BYTE_ARRAY_OFFSET + arrIndex,
+ numBytes);
+ } else {
+ heapWriteCharsUTF16BE(chars, arrIndex, numBytes, targetArray);
+ }
+ } else {
+ // TODO: off-heap
+ }
+ buffer._unsafeWriterIndex(writerIndex);
+ }
+
+ public void writeCharsUTF8(MemoryBuffer buffer, char[] chars) {
+ int estimateMaxBytes = chars.length * 3;
+ int writerIndex = buffer.writerIndex();
+ buffer.ensure(writerIndex + 9 + estimateMaxBytes);
+ byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int arrIndex = buffer._unsafeHeapWriterIndex();
+ int targetIndex = fastEncodeUTF8(chars, targetArray, arrIndex + 9);
+ int written = targetIndex - arrIndex - 9;
+ buffer.putByte(arrIndex, UTF8);
+ buffer.putInt32(arrIndex + 1, chars.length << 1);
+ buffer.putInt32(arrIndex + 5, written);
+ buffer._unsafeWriterIndex(targetIndex);
+ } else {
+ // TODO: off-heap
+ }
+ }
+
+ public void writeBytesUTF8(MemoryBuffer buffer, byte[] bytes) {
+ int estimateMaxBytes = bytes.length / 2 * 3;
+ int writerIndex = buffer.writerIndex();
+ buffer.ensure(writerIndex + 9 + estimateMaxBytes);
+ byte[] targetArray = buffer.getHeapMemory();
+ if (targetArray != null) {
+ int arrIndex = buffer._unsafeHeapWriterIndex();
+ int targetIndex = fastEncodeUTF8(bytes, targetArray, arrIndex + 9);
+ int written = targetIndex - arrIndex - 9;
+ buffer.putByte(arrIndex, UTF8);
+ buffer.putInt32(arrIndex + 1, bytes.length);
+ buffer.putInt32(arrIndex + 5, written);
+ buffer._unsafeWriterIndex(targetIndex);
+ }
+ }
+
+ private static boolean fastDecodeUTF8(byte[] src, int offset, int len, byte[] dst) {
+ final int end = offset + len;
+ int dp = 0;
+
+ while (offset < end) {
+ if (offset + 8 <= end
+ && (Platform.getLong(src, Platform.BYTE_ARRAY_OFFSET + offset) & 0x8080808080808080L)
+ == 0) {
+ // ascii only
+ for (int i = 0, pos = Platform.IS_LITTLE_ENDIAN ? dp : dp + 1; i < 8; ++i, pos += 2) {
Review Comment:
Yes, I checked the assembly code and it is indeed not loop-unrolled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] [WIP] string optimize poc [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1805957168
##########
java/fury-core/src/main/java/org/apache/fury/serializer/StringSerializer.java:
##########
@@ -609,6 +638,95 @@ public void writeUTF8String(MemoryBuffer buffer, String value) {
buffer.writeBytes(bytes);
}
+ public static byte bestCoder(char[] chars) {
+ int numChars = chars.length;
+ int vectorizedLen = numChars >> 2;
+ int vectorizedChars = vectorizedLen << 2;
+ // sample 64 chars
+ int sampleNum = Math.min(64, numChars);
+ int endOffset =
+ Math.min(
+ Platform.CHAR_ARRAY_OFFSET + (vectorizedChars << 1),
+ Platform.CHAR_ARRAY_OFFSET + (sampleNum << 1));
+ int count = 0;
+ for (int offset = Platform.CHAR_ARRAY_OFFSET, charOffset = 0;
+ offset < endOffset;
+ offset += 8, charOffset += 4) {
+ long multiChars = Platform.getLong(chars, offset);
+ if ((multiChars & MULTI_CHARS_NON_LATIN_MASK) == 0) {
+ count += 4;
+ } else {
+ for (int i = 0; i < 4; ++i) {
+ if (chars[charOffset + i] < 0x80) {
+ count++;
+ }
+ }
+ }
+ }
+ // ascii number > 50%, choose UTF-8
+ if (count >= sampleNum * 0.5) {
+ return UTF8;
+ } else {
+ return UTF16;
+ }
+ }
+
+ public static byte bestCoder(byte[] bytes) {
+ int numBytes = bytes.length;
+ int vectorizedLen = numBytes >> 3;
+ int vectorizedBytes = vectorizedLen << 3;
+ // sample 64 chars
+ int sampleNum = Math.min(64 << 1, numBytes);
+ int endOffset =
+ Math.min(
+ Platform.BYTE_ARRAY_OFFSET + vectorizedBytes, Platform.BYTE_ARRAY_OFFSET + sampleNum);
+ int count = 0;
+ for (int offset = Platform.BYTE_ARRAY_OFFSET; offset < endOffset; offset += 8) {
+ long multiChars = Platform.getLong(bytes, offset);
+ if ((multiChars & MULTI_CHARS_NON_LATIN_MASK) == 0) {
+ count += 4;
+ } else {
+ for (int i = Platform.IS_LITTLE_ENDIAN ? 1 : 0; i < 8; ++i) {
+ if (bytes[offset + i] == 0) {
+ count++;
+ }
+ }
+ }
+ }
+ // ascii number > 50%, choose UTF-8
+ if (count >= sampleNum * 0.5) {
+ return UTF8;
+ } else {
+ return UTF16;
+ }
+ }
+
+ public void fastWriteString(MemoryBuffer buffer, String value) {
+ if (STRING_VALUE_FIELD_IS_BYTES) {
+ final byte[] bytes = (byte[]) Platform.getObject(value, STRING_VALUE_FIELD_OFFSET);
+ final byte coder = Platform.getByte(value, Offset.STRING_CODER_FIELD_OFFSET);
+ if (coder == LATIN1) {
+ // LATIN1
+ writeBytesLatin1(buffer, bytes);
+ } else {
+ if (bestCoder(bytes) == UTF8) {
+ writeBytesUTF8(buffer, bytes);
+ } else {
+ writeBytesUTF16(buffer, bytes);
+ }
+ }
+ } else {
+ final char[] chars = (char[]) Platform.getObject(value, STRING_VALUE_FIELD_OFFSET);
+ if (StringUtils.isLatin(chars)) {
Review Comment:
Maybe we could check latin1 in bestCoder for chars
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): Refactor String serialization and deserialization [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1807805720
##########
java/fury-core/src/main/java/org/apache/fury/util/StringEncodingUtils.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fury.util;
+
+import static org.apache.fury.util.StringUtils.MULTI_CHARS_NON_LATIN_MASK;
+
+import org.apache.fury.memory.Platform;
+
+/** String Encoding Utils. */
+public class StringEncodingUtils {
+
+ public static int convertUTF16ToUTF8(char[] src, byte[] dst, int dp) {
+ int numChars = src.length;
+ for (int charOffset = 0; charOffset < numChars; ) {
+ if (charOffset + 4 <= numChars
+ && (Platform.getLong(src, Platform.CHAR_ARRAY_OFFSET + charOffset * 2L)
+ & MULTI_CHARS_NON_LATIN_MASK)
+ == 0) {
+ // ascii only
+ dst[dp] = (byte) src[charOffset];
+ dst[dp + 1] = (byte) src[charOffset + 1];
+ dst[dp + 2] = (byte) src[charOffset + 2];
+ dst[dp + 3] = (byte) src[charOffset + 3];
+ dp += 4;
+ charOffset += 4;
+ } else {
+ char c = src[charOffset++];
+ if (c < 0x80) {
+ dst[dp++] = (byte) c;
+ } else if (c < 0x800) {
+ dst[dp] = (byte) (0xc0 | (c >> 6));
+ dst[dp + 1] = (byte) (0x80 | (c & 0x3f));
+ dp += 2;
+ } else if (c >= '\uD800' && c <= Character.MAX_LOW_SURROGATE) {
Review Comment:
I'm not sure which will be faster, maybe need some benchmark. Could you use MediaContent for benchmark?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): Refactor String serialization and deserialization [fury]
Posted by "HuangXingBo (via GitHub)" <gi...@apache.org>.
HuangXingBo commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1807877155
##########
java/fury-core/src/main/java/org/apache/fury/util/StringEncodingUtils.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fury.util;
+
+import static org.apache.fury.util.StringUtils.MULTI_CHARS_NON_LATIN_MASK;
+
+import org.apache.fury.memory.Platform;
+
+/** String Encoding Utils. */
+public class StringEncodingUtils {
+
+ public static int convertUTF16ToUTF8(char[] src, byte[] dst, int dp) {
+ int numChars = src.length;
+ for (int charOffset = 0; charOffset < numChars; ) {
+ if (charOffset + 4 <= numChars
+ && (Platform.getLong(src, Platform.CHAR_ARRAY_OFFSET + charOffset * 2L)
+ & MULTI_CHARS_NON_LATIN_MASK)
+ == 0) {
+ // ascii only
+ dst[dp] = (byte) src[charOffset];
+ dst[dp + 1] = (byte) src[charOffset + 1];
+ dst[dp + 2] = (byte) src[charOffset + 2];
+ dst[dp + 3] = (byte) src[charOffset + 3];
+ dp += 4;
+ charOffset += 4;
+ } else {
+ char c = src[charOffset++];
+ if (c < 0x80) {
+ dst[dp++] = (byte) c;
+ } else if (c < 0x800) {
+ dst[dp] = (byte) (0xc0 | (c >> 6));
+ dst[dp + 1] = (byte) (0x80 | (c & 0x3f));
+ dp += 2;
+ } else if (c >= '\uD800' && c <= Character.MAX_LOW_SURROGATE) {
Review Comment:
This branch can be extracted, but I think for this kind of for loop with complex logic, iniline or not is not so critical, and even forcing inline of overly complex code can be a side effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] [WIP] string optimize poc [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on PR #1890:
URL: https://github.com/apache/fury/pull/1890#issuecomment-2421554822
And maybe we should provide a StringEncoding option too:
```
enum StingEncoding {
NATIVE_STRING,
LATIN1_UTF16,
LATIN1_UTF8_16,
}
```
Sometimes big serialized binary is accepptable to users
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): Refactor String serialization and deserialization [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1807807093
##########
java/fury-core/src/main/java/org/apache/fury/util/StringEncodingUtils.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fury.util;
+
+import static org.apache.fury.util.StringUtils.MULTI_CHARS_NON_LATIN_MASK;
+
+import org.apache.fury.memory.Platform;
+
+/** String Encoding Utils. */
+public class StringEncodingUtils {
+
Review Comment:
```suggestion
/**
* A fast encoding algorithm to convert a utf16 encoded string into utf8 encoded
* string when most chars in the string are ascii chars.
*/
```
##########
java/fury-core/src/main/java/org/apache/fury/util/StringEncodingUtils.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fury.util;
+
+import static org.apache.fury.util.StringUtils.MULTI_CHARS_NON_LATIN_MASK;
+
+import org.apache.fury.memory.Platform;
+
+/** String Encoding Utils. */
+public class StringEncodingUtils {
+
+ public static int convertUTF16ToUTF8(char[] src, byte[] dst, int dp) {
+ int numChars = src.length;
+ for (int charOffset = 0; charOffset < numChars; ) {
+ if (charOffset + 4 <= numChars
+ && (Platform.getLong(src, Platform.CHAR_ARRAY_OFFSET + charOffset * 2L)
+ & MULTI_CHARS_NON_LATIN_MASK)
+ == 0) {
+ // ascii only
+ dst[dp] = (byte) src[charOffset];
+ dst[dp + 1] = (byte) src[charOffset + 1];
+ dst[dp + 2] = (byte) src[charOffset + 2];
+ dst[dp + 3] = (byte) src[charOffset + 3];
+ dp += 4;
+ charOffset += 4;
+ } else {
+ char c = src[charOffset++];
+ if (c < 0x80) {
+ dst[dp++] = (byte) c;
+ } else if (c < 0x800) {
+ dst[dp] = (byte) (0xc0 | (c >> 6));
+ dst[dp + 1] = (byte) (0x80 | (c & 0x3f));
+ dp += 2;
+ } else if (c >= '\uD800' && c <= Character.MAX_LOW_SURROGATE) {
+ char d;
+ if (c > Character.MAX_HIGH_SURROGATE
+ || charOffset == src.length
+ || (d = src[charOffset]) < Character.MIN_LOW_SURROGATE
+ || d > Character.MAX_LOW_SURROGATE) {
+ throw new RuntimeException("malformed input off : " + charOffset);
+ }
+
+ int uc = ((c << 10) + d) + (0x010000 - ('\uD800' << 10) - Character.MIN_LOW_SURROGATE);
+ dst[dp] = (byte) (0xf0 | ((uc >> 18)));
+ dst[dp + 1] = (byte) (0x80 | ((uc >> 12) & 0x3f));
+ dst[dp + 2] = (byte) (0x80 | ((uc >> 6) & 0x3f));
+ dst[dp + 3] = (byte) (0x80 | (uc & 0x3f));
+ dp += 4;
+ charOffset++;
+ } else {
+ dst[dp] = (byte) (0xe0 | ((c >> 12)));
+ dst[dp + 1] = (byte) (0x80 | ((c >> 6) & 0x3f));
+ dst[dp + 2] = (byte) (0x80 | (c & 0x3f));
+ dp += 3;
+ }
+ }
+ }
+ return dp;
+ }
+
Review Comment:
```suggestion
/**
* A fast decoding algorithm to convert a utf8 encoded string into utf16 encoded
* string when most chars in the string are ascii chars.
*/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): Refactor String serialization and deserialization [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1807806539
##########
java/fury-core/src/main/java/org/apache/fury/util/StringEncodingUtils.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fury.util;
+
+import static org.apache.fury.util.StringUtils.MULTI_CHARS_NON_LATIN_MASK;
+
+import org.apache.fury.memory.Platform;
+
+/** String Encoding Utils. */
+public class StringEncodingUtils {
+
+ public static int convertUTF16ToUTF8(char[] src, byte[] dst, int dp) {
Review Comment:
Maybe we need to add a javadoc to document that this method will be faster only when there are many assci chars in the string. Acutally I'm curious whether this method will be faster than JDK native implementation for other string
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): Refactor String serialization and deserialization [fury]
Posted by "HuangXingBo (via GitHub)" <gi...@apache.org>.
HuangXingBo commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1807872914
##########
java/fury-core/src/main/java/org/apache/fury/util/StringEncodingUtils.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fury.util;
+
+import static org.apache.fury.util.StringUtils.MULTI_CHARS_NON_LATIN_MASK;
+
+import org.apache.fury.memory.Platform;
+
+/** String Encoding Utils. */
+public class StringEncodingUtils {
+
+ public static int convertUTF16ToUTF8(char[] src, byte[] dst, int dp) {
Review Comment:
The overall algorithm idea implemented by this method is based on the scalar version of simdutf, and the non-ascillary part is based on the implementation of fastjson2(The speedup ratio is twice of JDK). Even in the all-Chinese scenario, it will be faster than the JDK implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] [WIP] string optimize poc [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on PR #1890:
URL: https://github.com/apache/fury/pull/1890#issuecomment-2421556012
Another thing is that `org.apache.fury.serializer.StringSerializer#writeStringExpr/readStringExpr` should also be updated to pick up right encoding
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] [WIP] string optimize poc [fury]
Posted by "HuangXingBo (via GitHub)" <gi...@apache.org>.
HuangXingBo commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1806617069
##########
java/fury-core/src/main/java/org/apache/fury/serializer/StringSerializer.java:
##########
@@ -609,6 +638,95 @@ public void writeUTF8String(MemoryBuffer buffer, String value) {
buffer.writeBytes(bytes);
}
+ public static byte bestCoder(char[] chars) {
+ int numChars = chars.length;
+ int vectorizedLen = numChars >> 2;
+ int vectorizedChars = vectorizedLen << 2;
+ // sample 64 chars
+ int sampleNum = Math.min(64, numChars);
+ int endOffset =
+ Math.min(
+ Platform.CHAR_ARRAY_OFFSET + (vectorizedChars << 1),
+ Platform.CHAR_ARRAY_OFFSET + (sampleNum << 1));
+ int count = 0;
+ for (int offset = Platform.CHAR_ARRAY_OFFSET, charOffset = 0;
+ offset < endOffset;
+ offset += 8, charOffset += 4) {
+ long multiChars = Platform.getLong(chars, offset);
+ if ((multiChars & MULTI_CHARS_NON_LATIN_MASK) == 0) {
+ count += 4;
+ } else {
+ for (int i = 0; i < 4; ++i) {
+ if (chars[charOffset + i] < 0x80) {
+ count++;
+ }
+ }
+ }
+ }
+ // ascii number > 50%, choose UTF-8
+ if (count >= sampleNum * 0.5) {
+ return UTF8;
+ } else {
+ return UTF16;
+ }
+ }
+
+ public static byte bestCoder(byte[] bytes) {
+ int numBytes = bytes.length;
+ int vectorizedLen = numBytes >> 3;
+ int vectorizedBytes = vectorizedLen << 3;
+ // sample 64 chars
+ int sampleNum = Math.min(64 << 1, numBytes);
+ int endOffset =
+ Math.min(
+ Platform.BYTE_ARRAY_OFFSET + vectorizedBytes, Platform.BYTE_ARRAY_OFFSET + sampleNum);
+ int count = 0;
+ for (int offset = Platform.BYTE_ARRAY_OFFSET; offset < endOffset; offset += 8) {
+ long multiChars = Platform.getLong(bytes, offset);
+ if ((multiChars & MULTI_CHARS_NON_LATIN_MASK) == 0) {
+ count += 4;
+ } else {
+ for (int i = Platform.IS_LITTLE_ENDIAN ? 1 : 0; i < 8; ++i) {
+ if (bytes[offset + i] == 0) {
+ count++;
+ }
+ }
+ }
+ }
+ // ascii number > 50%, choose UTF-8
+ if (count >= sampleNum * 0.5) {
+ return UTF8;
+ } else {
+ return UTF16;
+ }
+ }
+
+ public void fastWriteString(MemoryBuffer buffer, String value) {
+ if (STRING_VALUE_FIELD_IS_BYTES) {
+ final byte[] bytes = (byte[]) Platform.getObject(value, STRING_VALUE_FIELD_OFFSET);
+ final byte coder = Platform.getByte(value, Offset.STRING_CODER_FIELD_OFFSET);
+ if (coder == LATIN1) {
+ // LATIN1
+ writeBytesLatin1(buffer, bytes);
+ } else {
+ if (bestCoder(bytes) == UTF8) {
+ writeBytesUTF8(buffer, bytes);
+ } else {
+ writeBytesUTF16(buffer, bytes);
+ }
+ }
+ } else {
+ final char[] chars = (char[]) Platform.getObject(value, STRING_VALUE_FIELD_OFFSET);
+ if (StringUtils.isLatin(chars)) {
Review Comment:
Yes. it is a good suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] [WIP] string optimize poc [fury]
Posted by "HuangXingBo (via GitHub)" <gi...@apache.org>.
HuangXingBo commented on PR #1890:
URL: https://github.com/apache/fury/pull/1890#issuecomment-2422642573
@chaokunyang Thanks a lot for the comments. I think it is sufficient to continue using the `isCompress` configuration, which is relatively easy to understand for users. Introducing a new encoding configurations are difficult for users to understand. I will make changes based on the comments and submit a formal PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] [WIP] string optimize poc [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1805964865
##########
java/fury-core/src/main/java/org/apache/fury/serializer/StringSerializer.java:
##########
@@ -609,6 +638,95 @@ public void writeUTF8String(MemoryBuffer buffer, String value) {
buffer.writeBytes(bytes);
}
+ public static byte bestCoder(char[] chars) {
+ int numChars = chars.length;
+ int vectorizedLen = numChars >> 2;
+ int vectorizedChars = vectorizedLen << 2;
+ // sample 64 chars
+ int sampleNum = Math.min(64, numChars);
+ int endOffset =
+ Math.min(
+ Platform.CHAR_ARRAY_OFFSET + (vectorizedChars << 1),
+ Platform.CHAR_ARRAY_OFFSET + (sampleNum << 1));
+ int count = 0;
+ for (int offset = Platform.CHAR_ARRAY_OFFSET, charOffset = 0;
+ offset < endOffset;
+ offset += 8, charOffset += 4) {
+ long multiChars = Platform.getLong(chars, offset);
+ if ((multiChars & MULTI_CHARS_NON_LATIN_MASK) == 0) {
+ count += 4;
+ } else {
+ for (int i = 0; i < 4; ++i) {
+ if (chars[charOffset + i] < 0x80) {
+ count++;
+ }
+ }
+ }
+ }
+ // ascii number > 50%, choose UTF-8
+ if (count >= sampleNum * 0.5) {
+ return UTF8;
+ } else {
+ return UTF16;
+ }
+ }
+
+ public static byte bestCoder(byte[] bytes) {
+ int numBytes = bytes.length;
+ int vectorizedLen = numBytes >> 3;
+ int vectorizedBytes = vectorizedLen << 3;
+ // sample 64 chars
+ int sampleNum = Math.min(64 << 1, numBytes);
+ int endOffset =
+ Math.min(
+ Platform.BYTE_ARRAY_OFFSET + vectorizedBytes, Platform.BYTE_ARRAY_OFFSET + sampleNum);
+ int count = 0;
+ for (int offset = Platform.BYTE_ARRAY_OFFSET; offset < endOffset; offset += 8) {
+ long multiChars = Platform.getLong(bytes, offset);
+ if ((multiChars & MULTI_CHARS_NON_LATIN_MASK) == 0) {
+ count += 4;
+ } else {
+ for (int i = Platform.IS_LITTLE_ENDIAN ? 1 : 0; i < 8; ++i) {
Review Comment:
I'm not sure whether unrolling will be faster, maybe need a benchmark. One drawback is unrolling will make this method have bigger bytecode size. But if it's faster, we could afford that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): Refactor String serialization and deserialization [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1807807425
##########
java/fury-core/src/main/java/org/apache/fury/util/StringEncodingUtils.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fury.util;
+
+import static org.apache.fury.util.StringUtils.MULTI_CHARS_NON_LATIN_MASK;
+
+import org.apache.fury.memory.Platform;
+
+/** String Encoding Utils. */
+public class StringEncodingUtils {
+
+ public static int convertUTF16ToUTF8(char[] src, byte[] dst, int dp) {
+ int numChars = src.length;
+ for (int charOffset = 0; charOffset < numChars; ) {
+ if (charOffset + 4 <= numChars
+ && (Platform.getLong(src, Platform.CHAR_ARRAY_OFFSET + charOffset * 2L)
+ & MULTI_CHARS_NON_LATIN_MASK)
+ == 0) {
+ // ascii only
+ dst[dp] = (byte) src[charOffset];
+ dst[dp + 1] = (byte) src[charOffset + 1];
+ dst[dp + 2] = (byte) src[charOffset + 2];
+ dst[dp + 3] = (byte) src[charOffset + 3];
+ dp += 4;
+ charOffset += 4;
+ } else {
+ char c = src[charOffset++];
+ if (c < 0x80) {
+ dst[dp++] = (byte) c;
+ } else if (c < 0x800) {
+ dst[dp] = (byte) (0xc0 | (c >> 6));
+ dst[dp + 1] = (byte) (0x80 | (c & 0x3f));
+ dp += 2;
+ } else if (c >= '\uD800' && c <= Character.MAX_LOW_SURROGATE) {
+ char d;
+ if (c > Character.MAX_HIGH_SURROGATE
+ || charOffset == src.length
+ || (d = src[charOffset]) < Character.MIN_LOW_SURROGATE
+ || d > Character.MAX_LOW_SURROGATE) {
+ throw new RuntimeException("malformed input off : " + charOffset);
+ }
+
+ int uc = ((c << 10) + d) + (0x010000 - ('\uD800' << 10) - Character.MIN_LOW_SURROGATE);
+ dst[dp] = (byte) (0xf0 | ((uc >> 18)));
+ dst[dp + 1] = (byte) (0x80 | ((uc >> 12) & 0x3f));
+ dst[dp + 2] = (byte) (0x80 | ((uc >> 6) & 0x3f));
+ dst[dp + 3] = (byte) (0x80 | (uc & 0x3f));
+ dp += 4;
+ charOffset++;
+ } else {
+ dst[dp] = (byte) (0xe0 | ((c >> 12)));
+ dst[dp + 1] = (byte) (0x80 | ((c >> 6) & 0x3f));
+ dst[dp + 2] = (byte) (0x80 | (c & 0x3f));
+ dp += 3;
+ }
+ }
+ }
+ return dp;
+ }
+
Review Comment:
```suggestion
/**
* A fast decoding implementation to convert a utf8 encoded string into utf16 encoded
* string when most chars in the string are ascii chars.
*/
```
##########
java/fury-core/src/main/java/org/apache/fury/util/StringEncodingUtils.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fury.util;
+
+import static org.apache.fury.util.StringUtils.MULTI_CHARS_NON_LATIN_MASK;
+
+import org.apache.fury.memory.Platform;
+
+/** String Encoding Utils. */
+public class StringEncodingUtils {
+
+ public static int convertUTF16ToUTF8(char[] src, byte[] dst, int dp) {
+ int numChars = src.length;
+ for (int charOffset = 0; charOffset < numChars; ) {
+ if (charOffset + 4 <= numChars
+ && (Platform.getLong(src, Platform.CHAR_ARRAY_OFFSET + charOffset * 2L)
+ & MULTI_CHARS_NON_LATIN_MASK)
+ == 0) {
+ // ascii only
+ dst[dp] = (byte) src[charOffset];
+ dst[dp + 1] = (byte) src[charOffset + 1];
+ dst[dp + 2] = (byte) src[charOffset + 2];
+ dst[dp + 3] = (byte) src[charOffset + 3];
+ dp += 4;
+ charOffset += 4;
+ } else {
+ char c = src[charOffset++];
+ if (c < 0x80) {
+ dst[dp++] = (byte) c;
+ } else if (c < 0x800) {
+ dst[dp] = (byte) (0xc0 | (c >> 6));
+ dst[dp + 1] = (byte) (0x80 | (c & 0x3f));
+ dp += 2;
+ } else if (c >= '\uD800' && c <= Character.MAX_LOW_SURROGATE) {
+ char d;
+ if (c > Character.MAX_HIGH_SURROGATE
+ || charOffset == src.length
+ || (d = src[charOffset]) < Character.MIN_LOW_SURROGATE
+ || d > Character.MAX_LOW_SURROGATE) {
+ throw new RuntimeException("malformed input off : " + charOffset);
+ }
+
+ int uc = ((c << 10) + d) + (0x010000 - ('\uD800' << 10) - Character.MIN_LOW_SURROGATE);
+ dst[dp] = (byte) (0xf0 | ((uc >> 18)));
+ dst[dp + 1] = (byte) (0x80 | ((uc >> 12) & 0x3f));
+ dst[dp + 2] = (byte) (0x80 | ((uc >> 6) & 0x3f));
+ dst[dp + 3] = (byte) (0x80 | (uc & 0x3f));
+ dp += 4;
+ charOffset++;
+ } else {
+ dst[dp] = (byte) (0xe0 | ((c >> 12)));
+ dst[dp + 1] = (byte) (0x80 | ((c >> 6) & 0x3f));
+ dst[dp + 2] = (byte) (0x80 | (c & 0x3f));
+ dp += 3;
+ }
+ }
+ }
+ return dp;
+ }
+
Review Comment:
```suggestion
/**
* A fast decoding implementation to convert a utf8 encoded string into utf16 encoded
* string when most chars in the string are ascii chars.
*/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): Refactor String serialization and deserialization [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1807805375
##########
java/fury-core/src/main/java/org/apache/fury/util/StringEncodingUtils.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fury.util;
+
+import static org.apache.fury.util.StringUtils.MULTI_CHARS_NON_LATIN_MASK;
+
+import org.apache.fury.memory.Platform;
+
+/** String Encoding Utils. */
+public class StringEncodingUtils {
+
+ public static int convertUTF16ToUTF8(char[] src, byte[] dst, int dp) {
+ int numChars = src.length;
+ for (int charOffset = 0; charOffset < numChars; ) {
+ if (charOffset + 4 <= numChars
+ && (Platform.getLong(src, Platform.CHAR_ARRAY_OFFSET + charOffset * 2L)
+ & MULTI_CHARS_NON_LATIN_MASK)
+ == 0) {
+ // ascii only
+ dst[dp] = (byte) src[charOffset];
+ dst[dp + 1] = (byte) src[charOffset + 1];
+ dst[dp + 2] = (byte) src[charOffset + 2];
+ dst[dp + 3] = (byte) src[charOffset + 3];
+ dp += 4;
+ charOffset += 4;
+ } else {
+ char c = src[charOffset++];
+ if (c < 0x80) {
+ dst[dp++] = (byte) c;
+ } else if (c < 0x800) {
+ dst[dp] = (byte) (0xc0 | (c >> 6));
+ dst[dp + 1] = (byte) (0x80 | (c & 0x3f));
+ dp += 2;
+ } else if (c >= '\uD800' && c <= Character.MAX_LOW_SURROGATE) {
Review Comment:
How big `convertUTF16ToUTF8`? Can this method be inlined to its caller?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): Refactor String serialization and deserialization [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1807807093
##########
java/fury-core/src/main/java/org/apache/fury/util/StringEncodingUtils.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fury.util;
+
+import static org.apache.fury.util.StringUtils.MULTI_CHARS_NON_LATIN_MASK;
+
+import org.apache.fury.memory.Platform;
+
+/** String Encoding Utils. */
+public class StringEncodingUtils {
+
Review Comment:
```suggestion
/**
* A fast encoding implementation to convert a utf16 encoded string into utf8 encoded
* binary when most chars in the string are ascii chars.
*/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): Refactor String serialization and deserialization [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #1890:
URL: https://github.com/apache/fury/pull/1890#discussion_r1808628816
##########
java/fury-core/src/main/java/org/apache/fury/util/StringEncodingUtils.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fury.util;
+
+import static org.apache.fury.util.StringUtils.MULTI_CHARS_NON_LATIN_MASK;
+
+import org.apache.fury.memory.Platform;
+
+/** String Encoding Utils. */
+public class StringEncodingUtils {
+
+ public static int convertUTF16ToUTF8(char[] src, byte[] dst, int dp) {
+ int numChars = src.length;
+ for (int charOffset = 0; charOffset < numChars; ) {
+ if (charOffset + 4 <= numChars
+ && (Platform.getLong(src, Platform.CHAR_ARRAY_OFFSET + charOffset * 2L)
+ & MULTI_CHARS_NON_LATIN_MASK)
+ == 0) {
+ // ascii only
+ dst[dp] = (byte) src[charOffset];
+ dst[dp + 1] = (byte) src[charOffset + 1];
+ dst[dp + 2] = (byte) src[charOffset + 2];
+ dst[dp + 3] = (byte) src[charOffset + 3];
+ dp += 4;
+ charOffset += 4;
+ } else {
+ char c = src[charOffset++];
+ if (c < 0x80) {
+ dst[dp++] = (byte) c;
+ } else if (c < 0x800) {
+ dst[dp] = (byte) (0xc0 | (c >> 6));
+ dst[dp + 1] = (byte) (0x80 | (c & 0x3f));
+ dp += 2;
+ } else if (c >= '\uD800' && c <= Character.MAX_LOW_SURROGATE) {
Review Comment:
I see, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org