You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fury.apache.org by ch...@apache.org on 2024/07/08 13:40:42 UTC

(fury) branch main updated: feat(Python): meta string encoding algorithm in Python (#1702)

This is an automated email from the ASF dual-hosted git repository.

chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/fury.git


The following commit(s) were added to refs/heads/main by this push:
     new 15e25bde feat(Python): meta string encoding algorithm in Python (#1702)
15e25bde is described below

commit 15e25bde36957a7b61b475c74c47cf95e16ea46d
Author: PAN <46...@users.noreply.github.com>
AuthorDate: Mon Jul 8 21:40:38 2024 +0800

    feat(Python): meta string encoding algorithm in Python (#1702)
    
    <!--
    **Thanks for contributing to Fury.**
    
    **If this is your first time opening a PR on fury, you can refer to
    [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md).**
    
    Contribution Checklist
    
    - The **Apache Fury (incubating)** community has restrictions on the
    naming of pr titles. You can also find instructions in
    [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md).
    
    - Fury has a strong focus on performance. If the PR you submit will have
    an impact on performance, please benchmark it first and provide the
    benchmark result here.
    -->
    
    ## What does this PR do?
    The Python algorithm for meta string is implemented and has passed all
    13 tests
    <img width="893" alt="image"
    src="https://github.com/apache/fury/assets/46820719/0a23c236-8fe1-4a31-9d44-263982f5c15c">
    
    I am not familiar with Python, there may be a few problems, but the
    overall code is relatively complete
    
    How to have a better implementation, I would also like to improve it, I
    like this projectBut I think this test environment is relatively bad,
    and I feel that it can be further built
    
    <!-- Describe the purpose of this PR. -->
    
    
    ## Related issues
    Closes #1541
    
    <!--
    Is there any related issue? Please attach here.
    
    - #xxxx0
    - #xxxx1
    - #xxxx2
    -->
    
    
    ## Does this PR introduce any user-facing change?
    
    <!--
    If any user-facing interface changes, please [open an
    issue](https://github.com/apache/fury/issues/new/choose) describing the
    need to do so and update the document if necessary.
    -->
    
    - [ ] Does this PR introduce any public API change?
    - [ ] Does this PR introduce any binary protocol compatibility change?
    
    
    ## Benchmark
    
    <!--
    When the PR has an impact on performance (if you don't know whether the
    PR will have an impact on performance, you can submit the PR first, and
    if it will have impact on performance, the code reviewer will explain
    it), be sure to attach a benchmark data here.
    -->
---
 python/pyfury/meta/__init__.py         |  16 ++
 python/pyfury/meta/metastring.py       | 510 +++++++++++++++++++++++++++++++++
 python/pyfury/tests/test_metastring.py | 193 +++++++++++++
 3 files changed, 719 insertions(+)

diff --git a/python/pyfury/meta/__init__.py b/python/pyfury/meta/__init__.py
new file mode 100644
index 00000000..13a83393
--- /dev/null
+++ b/python/pyfury/meta/__init__.py
@@ -0,0 +1,16 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/python/pyfury/meta/metastring.py b/python/pyfury/meta/metastring.py
new file mode 100644
index 00000000..63232b56
--- /dev/null
+++ b/python/pyfury/meta/metastring.py
@@ -0,0 +1,510 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from collections import namedtuple
+from enum import Enum
+from typing import List
+
+
+class Encoding(Enum):
+    """
+    Defines the types of supported encodings for MetaStrings.
+    """
+
+    UTF_8 = 0x00
+    LOWER_SPECIAL = 0x01
+    LOWER_UPPER_DIGIT_SPECIAL = 0x02
+    FIRST_TO_LOWER_SPECIAL = 0x03
+    ALL_TO_LOWER_SPECIAL = 0x04
+
+
+Statistics = namedtuple(
+    "Statistics",
+    [
+        "can_lower_upper_digit_special_encoded",
+        "can_lower_special_encoded",
+        "digit_count",
+        "upper_count",
+    ],
+)
+
+# _METASTRING_NUM_CHARS_LIMIT is used to check whether the length of the value is valid.
+_METASTRING_NUM_CHARS_LIMIT = 32767
+
+
+class MetaString:
+    def __init__(
+        self, original: str, encoding: Encoding, encoded_data: bytes, length: int
+    ):
+        self.original = original
+        self.encoding = encoding
+        self.encoded_data = encoded_data
+        self.length = length
+        if self.encoding != Encoding.UTF_8:
+            self.strip_last_char = (encoded_data[0] & 0x80) != 0
+        else:
+            self.strip_last_char = False
+
+
+class MetaStringDecoder:
+    """
+    Decodes MetaString objects back into their original plain text form.
+    """
+
+    def decode(self, encoded_data: bytes, encoding: Encoding) -> str:
+        """
+        Decodes the encoded data using the specified encoding.
+
+        Args:
+            encoded_data (bytes): The data to decode.
+            encoding (Encoding): The encoding type used for decoding.
+
+        Returns:
+            str: The decoded string.
+        """
+        if len(encoded_data) == 0:
+            return ""
+        return self.decode_with_encoding(encoded_data, encoding)
+
+    def decode_with_encoding(self, encoded_data: bytes, encoding: Encoding) -> str:
+        """
+        Decodes the encoded data with the specified encoding.
+
+        Args:
+            encoded_data (bytes): The data to decode.
+            encoding (Encoding): The encoding type.
+
+        Returns:
+            str: The decoded string.
+        """
+        if encoding == Encoding.LOWER_SPECIAL:
+            return self._decode_lower_special(encoded_data)
+        elif encoding == Encoding.LOWER_UPPER_DIGIT_SPECIAL:
+            return self._decode_lower_upper_digit_special(encoded_data)
+        elif encoding == Encoding.FIRST_TO_LOWER_SPECIAL:
+            return self._decode_rep_first_lower_special(encoded_data)
+        elif encoding == Encoding.ALL_TO_LOWER_SPECIAL:
+            return self._decode_rep_all_to_lower_special(encoded_data)
+        elif encoding == Encoding.UTF_8:
+            return encoded_data.decode("utf-8")
+        else:
+            raise ValueError(f"Unexpected encoding flag: {encoding}")
+
+    def _decode_lower_special(self, data: bytes) -> str:
+        """
+        Decodes data encoded with LOWER_SPECIAL encoding.
+
+        Args:
+            data (bytes): The encoded data.
+
+        Returns:
+            str: The decoded string.
+        """
+        decoded = []
+        num_bits = len(data) * 8  # Total number of bits in the data
+        strip_last_char = (data[0] & 0x80) != 0  # Check the first bit of the first byte
+        bit_index = 1
+        bit_mask = 0b11111
+        while bit_index + 5 <= num_bits and not (
+            strip_last_char and (bit_index + 2 * 5 > num_bits)
+        ):
+            byte_index = bit_index // 8
+            intra_byte_index = bit_index % 8
+            # Extract the 5-bit character value across byte boundaries if needed
+            char_value = (
+                (data[byte_index] << 8)
+                | (data[byte_index + 1] if byte_index + 1 < len(data) else 0)
+            ) >> (11 - intra_byte_index) & bit_mask
+            bit_index += 5
+            decoded.append(self._decode_lower_special_char(char_value))
+
+        return "".join(decoded)
+
+    def _decode_lower_upper_digit_special(self, data: bytes) -> str:
+        """
+        Decodes data encoded with LOWER_UPPER_DIGIT_SPECIAL encoding.
+
+        Args:
+            data (bytes): The encoded data.
+
+        Returns:
+            str: The decoded string.
+        """
+        decoded = []
+        bit_index = 1
+        strip_last_char = (data[0] & 0x80) != 0
+        bit_mask = 0b111111
+        num_bits = len(data) * 8
+        while bit_index + 6 <= num_bits and not (
+            strip_last_char and (bit_index + 2 * 6 > num_bits)
+        ):
+            byte_index = bit_index // 8
+            intra_byte_index = bit_index % 8
+            # Extract the 6-bit character value across byte boundaries if needed
+            char_value = (
+                (data[byte_index] << 8)
+                | (data[byte_index + 1] if byte_index + 1 < len(data) else 0)
+            ) >> (10 - intra_byte_index) & bit_mask
+            bit_index += 6
+            decoded.append(self._decode_lower_upper_digit_special_char(char_value))
+        return "".join(decoded)
+
+    def _decode_lower_special_char(self, char_value: int) -> str:
+        """
+        Decodes a single character encoded with LOWER_SPECIAL encoding.
+
+        Args:
+            char_value (int): The encoded character value.
+
+        Returns:
+            str: The decoded character.
+        """
+        if 0 <= char_value <= 25:
+            return chr(ord("a") + char_value)
+        elif char_value == 26:
+            return "."
+        elif char_value == 27:
+            return "_"
+        elif char_value == 28:
+            return "$"
+        elif char_value == 29:
+            return "|"
+        else:
+            raise ValueError(f"Invalid character value for LOWER_SPECIAL: {char_value}")
+
+    def _decode_lower_upper_digit_special_char(self, char_value: int) -> str:
+        """
+        Decodes a single character encoded with LOWER_UPPER_DIGIT_SPECIAL encoding.
+
+        Args:
+            char_value (int): The encoded character value.
+
+        Returns:
+            str: The decoded character.
+        """
+        if 0 <= char_value <= 25:
+            return chr(ord("a") + char_value)
+        elif 26 <= char_value <= 51:
+            return chr(ord("A") + (char_value - 26))
+        elif 52 <= char_value <= 61:
+            return chr(ord("0") + (char_value - 52))
+        elif char_value == 62:
+            return "."
+        elif char_value == 63:
+            return "_"
+        else:
+            raise ValueError(
+                f"Invalid character value for LOWER_UPPER_DIGIT_SPECIAL: {char_value}"
+            )
+
+    def _decode_rep_first_lower_special(self, data: bytes) -> str:
+        """
+        Decodes data encoded with FIRST_TO_LOWER_SPECIAL encoding.
+
+        Args:
+            data (bytes): The encoded data.
+
+        Returns:
+            str: The decoded string.
+        """
+        decoded_str = self._decode_lower_special(data)
+        return decoded_str.capitalize()
+
+    def _decode_rep_all_to_lower_special(self, data: bytes) -> str:
+        """
+        Decodes data encoded with ALL_TO_LOWER_SPECIAL encoding.
+
+        Args:
+            data (bytes): The encoded data.
+
+        Returns:
+            str: The decoded string.
+        """
+        decoded_str = self._decode_lower_special(data)
+        result = []
+        skip = False
+        for i, char in enumerate(decoded_str):
+            if skip:
+                skip = False
+                continue
+            if char == "|":
+                result.append(decoded_str[i + 1].upper())
+                skip = True
+            else:
+                result.append(char)
+        return "".join(result)
+
+
+class MetaStringEncoder:
+    """
+    Encodes plain text strings into MetaString objects with specified encoding mechanisms.
+    """
+
+    def encode(self, input_string: str) -> MetaString:
+        """
+        Encodes the input string into a MetaString object.
+
+        Args:
+            input_string (str): The string to encode.
+
+        Returns:
+            MetaString: The encoded MetaString object.
+        """
+        # Long meta string than _METASTRING_NUM_CHARS_LIMIT is not allowed.
+        assert (
+            len(input_string) < _METASTRING_NUM_CHARS_LIMIT
+        ), "Long meta string than _METASTRING_NUM_CHARS_LIMIT is not allowed."
+
+        if not input_string:
+            return MetaString(input_string, Encoding.UTF_8, bytes(), 0)
+
+        encoding = self.compute_encoding(input_string)
+        return self.encode_with_encoding(input_string, encoding)
+
+    def encode_with_encoding(self, input_string: str, encoding: Encoding) -> MetaString:
+        """
+        Encodes the input string with the specified encoding.
+
+        Args:
+            input_string (str): The string to encode.
+            encoding (Encoding): The encoding type.
+
+        Returns:
+            MetaString: An encoded MetaString object.
+        """
+        # Long meta string than _METASTRING_NUM_CHARS_LIMIT is not allowed.
+        assert (
+            len(input_string) < _METASTRING_NUM_CHARS_LIMIT
+        ), "Long meta string than _METASTRING_NUM_CHARS_LIMIT is not allowed."
+
+        if not input_string:
+            return MetaString(input_string, Encoding.UTF_8, bytes(), 0)
+
+        length = len(input_string)
+        if encoding == Encoding.LOWER_SPECIAL:
+            encoded_data = self._encode_lower_special(input_string)
+            return MetaString(input_string, encoding, encoded_data, length * 5)
+        elif encoding == Encoding.LOWER_UPPER_DIGIT_SPECIAL:
+            encoded_data = self._encode_lower_upper_digit_special(input_string)
+            return MetaString(input_string, encoding, encoded_data, length * 6)
+        elif encoding == Encoding.FIRST_TO_LOWER_SPECIAL:
+            encoded_data = self._encode_first_to_lower_special(input_string)
+            return MetaString(input_string, encoding, encoded_data, length * 5)
+        elif encoding == Encoding.ALL_TO_LOWER_SPECIAL:
+            chars = list(input_string)
+            upper_count = sum(1 for c in chars if c.isupper())
+            encoded_data = self._encode_all_to_lower_special(chars)
+            return MetaString(
+                input_string, encoding, encoded_data, (upper_count + length) * 5
+            )
+        else:
+            encoded_data = bytes(input_string, "utf-8")
+            return MetaString(
+                input_string, Encoding.UTF_8, encoded_data, len(encoded_data) * 8
+            )
+
+    def compute_encoding(self, input_string: str) -> Encoding:
+        """
+        Determines the encoding type of the input string.
+
+        Args:
+            input_string (str): The string to be encoded.
+
+        Returns:
+            Encoding: The encoding type.
+        """
+        if not input_string:
+            return Encoding.LOWER_SPECIAL
+
+        chars = list(input_string)
+        statistics = self._compute_statistics(chars)
+        if statistics.can_lower_special_encoded:
+            return Encoding.LOWER_SPECIAL
+        elif statistics.can_lower_upper_digit_special_encoded:
+            if statistics.digit_count != 0:
+                return Encoding.LOWER_UPPER_DIGIT_SPECIAL
+            else:
+                upper_count = statistics.upper_count
+                if upper_count == 1 and chars[0].isupper():
+                    return Encoding.FIRST_TO_LOWER_SPECIAL
+                if (len(chars) + upper_count) * 5 < len(chars) * 6:
+                    return Encoding.ALL_TO_LOWER_SPECIAL
+                else:
+                    return Encoding.LOWER_UPPER_DIGIT_SPECIAL
+        return Encoding.UTF_8
+
+    def _compute_statistics(self, chars: List[str]) -> Statistics:
+        """
+        Computes statistics for the given characters to determine encoding possibilities.
+
+        Args:
+            chars (list): List of characters to analyze.
+
+        Returns:
+            Statistics: A named tuple with encoding possibility flags and counts.
+        """
+        can_lower_upper_digit_special_encoded = True
+        can_lower_special_encoded = True
+        digit_count = 0
+        upper_count = 0
+        for c in chars:
+            if can_lower_upper_digit_special_encoded:
+                if not (c.islower() or c.isupper() or c.isdigit() or c in {".", "_"}):
+                    can_lower_upper_digit_special_encoded = False
+            if can_lower_special_encoded:
+                if not (c.islower() or c in {".", "_", "$", "|"}):
+                    can_lower_special_encoded = False
+            if c.isdigit():
+                digit_count += 1
+            if c.isupper():
+                upper_count += 1
+
+        return Statistics(
+            can_lower_upper_digit_special_encoded,
+            can_lower_special_encoded,
+            digit_count,
+            upper_count,
+        )
+
+    def _encode_lower_special(self, input_string: str) -> bytes:
+        """
+        Encodes the input string using LOWER_SPECIAL encoding.
+
+        Args:
+            input_string (str): The string to encode.
+
+        Returns:
+            bytes: The encoded data.
+        """
+        chars = list(input_string)
+        return self._encode_generic(chars, 5)
+
+    def _encode_lower_upper_digit_special(self, input_string: str) -> bytes:
+        """
+        Encodes the input string using LOWER_UPPER_DIGIT_SPECIAL encoding.
+
+        Args:
+            input_string (str): The string to encode.
+
+        Returns:
+            bytes: The encoded data.
+        """
+        chars = list(input_string)
+        return self._encode_generic(chars, 6)
+
+    def _encode_first_to_lower_special(self, input_string: str) -> bytes:
+        """
+        Encodes the input string using FIRST_TO_LOWER_SPECIAL encoding.
+
+        Args:
+            input_string (str): The string to encode.
+
+        Returns:
+            bytes: The encoded data.
+        """
+        chars = list(input_string)
+        chars[0] = chars[0].lower()
+        return self._encode_generic(chars, 5)
+
+    def _encode_all_to_lower_special(self, chars: List[str]) -> bytes:
+        """
+        Encodes the input string using ALL_TO_LOWER_SPECIAL encoding.
+
+        Args:
+            chars (list): The characters to encode.
+
+        Returns:
+            bytes: The encoded data.
+        """
+        new_chars = []
+        for c in chars:
+            if c.isupper():
+                new_chars.append("|")
+                new_chars.append(c.lower())
+            else:
+                new_chars.append(c)
+        return self._encode_generic(new_chars, 5)
+
+    def _encode_generic(self, chars: List[str], bits_per_char: int) -> bytes:
+        """
+        Generic encoding function for encoding characters into bytes.
+
+        Args:
+            chars (list): The characters to encode.
+            bits_per_char (int): The number of bits per character.
+
+        Returns:
+            bytes: The encoded data.
+        """
+        total_bits = len(chars) * bits_per_char + 1
+        byte_length = (total_bits + 7) // 8
+        bytes_array = bytearray(byte_length)
+        current_bit = 1
+        for c in chars:
+            value = self._char_to_value(c, bits_per_char)
+            for i in range(bits_per_char - 1, -1, -1):
+                if (value & (1 << i)) != 0:
+                    byte_pos = current_bit // 8
+                    bit_pos = current_bit % 8
+                    bytes_array[byte_pos] |= 1 << (7 - bit_pos)
+                current_bit += 1
+        strip_last_char = len(bytes_array) * 8 >= total_bits + bits_per_char
+        if strip_last_char:
+            bytes_array[0] = bytes_array[0] | 0x80
+        return bytes_array
+
+    def _char_to_value(self, c: str, bits_per_char: int) -> int:
+        """
+        Converts a character to its encoded value based on the number of bits per character.
+
+        Args:
+            c (str): The character to convert.
+            bits_per_char (int): The number of bits per character.
+
+        Returns:
+            int: The encoded value of the character.
+        """
+        if bits_per_char == 5:
+            if "a" <= c <= "z":
+                return ord(c) - ord("a")
+            elif c == ".":
+                return 26
+            elif c == "_":
+                return 27
+            elif c == "$":
+                return 28
+            elif c == "|":
+                return 29
+            else:
+                raise ValueError(
+                    f"Unsupported character for LOWER_SPECIAL encoding: {c}"
+                )
+        elif bits_per_char == 6:
+            if "a" <= c <= "z":
+                return ord(c) - ord("a")
+            elif "A" <= c <= "Z":
+                return 26 + (ord(c) - ord("A"))
+            elif "0" <= c <= "9":
+                return 52 + (ord(c) - ord("0"))
+            elif c == ".":
+                return 62
+            elif c == "_":
+                return 63
+            else:
+                raise ValueError(
+                    f"Unsupported character for LOWER_UPPER_DIGIT_SPECIAL encoding: {c}"
+                )
diff --git a/python/pyfury/tests/test_metastring.py b/python/pyfury/tests/test_metastring.py
new file mode 100644
index 00000000..7dd98ff7
--- /dev/null
+++ b/python/pyfury/tests/test_metastring.py
@@ -0,0 +1,193 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+from pyfury.meta.metastring import (
+    MetaStringEncoder,
+    MetaStringDecoder,
+    Encoding,
+)
+
+
+def test_encode_metastring_lower_special():
+    encoder = MetaStringEncoder()
+    decoder = MetaStringDecoder()
+    encoded = encoder._encode_lower_special("abc_def")
+    assert len(encoded) == 5
+    assert len(encoder.encode("org.apache.fury.benchmark.data").encoded_data) == 19
+    assert len(encoder.encode("MediaContent").encoded_data) == 9
+    decoded = decoder.decode(encoded, Encoding.LOWER_SPECIAL)
+    assert decoded == "abc_def"
+
+    for i in range(128):
+        builder = "".join(chr(ord("a") + j % 26) for j in range(i))
+        encoded = encoder._encode_lower_special(builder)
+        decoded = decoder.decode(encoded, Encoding.LOWER_SPECIAL)
+        assert decoded == builder
+
+
+def test_encode_metastring_lower_upper_digit_special():
+    encoder = MetaStringEncoder()
+    encoded = encoder._encode_lower_upper_digit_special("ExampleInput123")
+    assert len(encoded) == 12
+    decoder = MetaStringDecoder()
+    decoded = decoder.decode(encoded, Encoding.LOWER_UPPER_DIGIT_SPECIAL)
+    assert decoded == "ExampleInput123"
+
+    for i in range(1, 128):
+        string = create_string(i)
+        encoded = encoder._encode_lower_upper_digit_special(string)
+        decoded = decoder.decode(encoded, Encoding.LOWER_UPPER_DIGIT_SPECIAL)
+        assert decoded == string, f"Failed at {i}"
+
+
+def create_string(length):
+    chars = []
+    for j in range(length):
+        n = j % 64
+        if n < 26:
+            chars.append(chr(ord("a") + n))
+        elif n < 52:
+            chars.append(chr(ord("A") + (n - 26)))
+        elif n < 62:
+            chars.append(chr(ord("0") + (n - 52)))
+        elif n == 62:
+            chars.append(".")
+        else:
+            chars.append("_")
+    return "".join(chars)
+
+
+def test_metastring():
+
+    encoder = MetaStringEncoder()
+    for i in range(1, 128):
+        try:
+            string = create_string(i)
+            metastring = encoder.encode(string)
+            assert metastring.encoding != Encoding.UTF_8
+            assert metastring.original == string
+
+            decoder = MetaStringDecoder()
+            new_string = decoder.decode(metastring.encoded_data, metastring.encoding)
+            assert new_string == string
+        except Exception as e:
+            pytest.fail(f"Failed at {i} with exception: {str(e)}")
+
+
+def test_encode_empty_string():
+    encoder = MetaStringEncoder()
+    decoder = MetaStringDecoder()
+    for encoding in [
+        Encoding.LOWER_SPECIAL,
+        Encoding.LOWER_UPPER_DIGIT_SPECIAL,
+        Encoding.FIRST_TO_LOWER_SPECIAL,
+        Encoding.ALL_TO_LOWER_SPECIAL,
+        Encoding.UTF_8,
+    ]:
+        metastring = encoder.encode_with_encoding("", encoding)
+        assert len(metastring.encoded_data) == 0
+        decoded = decoder.decode(metastring.encoded_data, metastring.encoding)
+        assert decoded == ""
+
+
+def test_encode_characters_outside_of_lower_special():
+    encoder = MetaStringEncoder()
+
+    test_string = "abcdefABCDEF1234!@#"
+    metastring = encoder.encode(test_string)
+    assert metastring.encoding == Encoding.UTF_8
+
+
+def test_all_to_upper_special_encoding():
+    encoder = MetaStringEncoder()
+    decoder = MetaStringDecoder()
+    test_string = "ABC_DEF"
+    metastring = encoder.encode(test_string)
+    assert metastring.encoding == Encoding.LOWER_UPPER_DIGIT_SPECIAL
+    decoded_string = decoder.decode(metastring.encoded_data, metastring.encoding)
+    assert decoded_string == test_string
+
+
+def test_first_to_lower_special_encoding():
+    encoder = MetaStringEncoder()
+    decoder = MetaStringDecoder()
+    test_string = "Aabcdef"
+    metastring = encoder.encode(test_string)
+    assert metastring.encoding == Encoding.FIRST_TO_LOWER_SPECIAL
+    decoded_string = decoder.decode(metastring.encoded_data, metastring.encoding)
+    assert decoded_string == test_string
+
+
+def test_utf8_encoding():
+    encoder = MetaStringEncoder()
+    decoder = MetaStringDecoder()
+    test_string = "你好,世界"  # Non-Latin characters
+    metastring = encoder.encode(test_string)
+    assert metastring.encoding == Encoding.UTF_8
+    decoded_string = decoder.decode(metastring.encoded_data, metastring.encoding)
+    assert decoded_string == test_string
+
+
+def test_strip_last_char():
+    encoder = MetaStringEncoder()
+
+    test_string = "abc"  # encoded as 1|00000|00, 001|00010, exactly two bytes
+    encoded_metastring = encoder.encode(test_string)
+    assert not encoded_metastring.strip_last_char
+
+    test_string = "abcde"  # encoded as 1|00000|00, 001|00010, 00011|001, 00xxxxxx, stripped last char
+    encoded_metastring = encoder.encode(test_string)
+    assert encoded_metastring.strip_last_char
+
+
+def test_empty_string():
+    encoder = MetaStringEncoder()
+    decoder = MetaStringDecoder()
+    metastring = encoder.encode("")
+    assert metastring.encoded_data == bytes()
+
+    decoded = decoder.decode(metastring.encoded_data, metastring.encoding)
+    assert decoded == ""
+
+
+def test_ascii_encoding():
+    encoder = MetaStringEncoder()
+
+    test_string = "asciiOnly"
+    encoded_metastring = encoder.encode(test_string)
+    assert encoded_metastring.encoding != Encoding.UTF_8
+    assert encoded_metastring.encoding == Encoding.ALL_TO_LOWER_SPECIAL
+
+
+def test_non_ascii_encoding():
+    encoder = MetaStringEncoder()
+
+    test_string = "こんにちは"  # Non-ASCII string
+    encoded_metastring = encoder.encode(test_string)
+    assert encoded_metastring.encoding == Encoding.UTF_8
+
+
+def test_non_ascii_encoding_and_non_utf8():
+    encoder = MetaStringEncoder()
+
+    non_ascii_string = "こんにちは"  # Non-ASCII string
+
+    with pytest.raises(
+        ValueError, match="Unsupported character for LOWER_SPECIAL encoding: こ"
+    ):
+        encoder.encode_with_encoding(non_ascii_string, Encoding.LOWER_SPECIAL)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org