[I] [C++] Create fory::field<> template for field metadata [fory]

["chaokunyang (via GitHub)" <gi...@apache.org>]

chaokunyang opened a new issue, #3003:
URL: https://github.com/apache/fory/issues/3003

   ## Feature Request
   
   Create a `fory::field<>` template class for field metadata to enable performance and space optimization during xlang serialization.
   
   ## Is your feature request related to a problem? Please describe
   
   Currently, Fory's C++ xlang serialization treats all struct fields uniformly:
   1. **Null checks are always performed** - Even for fields that are never null, Fory writes a null/ref flag (1 byte per field)
   2. **Reference tracking is always applied** (when enabled globally) - Even for fields that won't be shared/cyclic, objects are tracked with hash lookup cost
   3. **Field names use meta string encoding** - In schema evolution mode, field names are encoded using meta string compression, but for fields with long names, this still takes space
   
   These defaults ensure correctness but introduce unnecessary overhead when the developer has more specific knowledge about their data model.
   
   ## Describe the solution you'd like
   
   Add a `fory::field<>` template in `field.h` that wraps field types with compile-time metadata:
   
   ```cpp
   #include <fory/serialization/field.h>
   #include <string>
   #include <memory>
   
   struct Foo {
       // Field f1: non-nullable (default), no ref tracking (default)
       // Tag ID 0 provides compact encoding in schema evolution mode
       fory::field<std::string, fory::id<0>> f1;
       
       // Field f2: non-nullable (default), no ref tracking (default)
       fory::field<Bar, fory::id<1>> f2;
       
       // Field f3: nullable field that may contain null values
       fory::field<std::optional<std::string>, fory::id<2>, fory::nullable> f3;
       
       // Field f4: shared reference that needs tracking (e.g., for circular refs)
       fory::field<std::shared_ptr<Node>, fory::id<3>, fory::ref, fory::nullable> parent;
       
       // Field with long name: tag ID provides significant space savings
       fory::field<std::string, fory::id<4>> very_long_field_name_that_would_take_many_bytes;
       
       // Explicit opt-out: use field name encoding but get nullable optimization
       fory::field<std::optional<std::string>, fory::id<-1>, fory::nullable> optional_field;
   };
   
   // Register with Fory
   FORY_REGISTER_TYPE(Foo);
   ```
   
   ### Template API Design
   
   ```cpp
   namespace fory {
   
   // Tag types for field properties
   template<int N>
   struct id { static constexpr int value = N; };
   
   struct nullable { static constexpr bool value = true; };
   struct ref { static constexpr bool value = true; };
   
   // Field wrapper template
   template<typename T, typename... Props>
   class field {
   public:
       using value_type = T;
       
       // Compile-time property extraction
       static constexpr int tag_id = /* extract from Props... */;
       static constexpr bool is_nullable = /* extract from Props... */;
       static constexpr bool track_ref = /* extract from Props... */;
       
       // Implicit conversion to/from T
       field() = default;
       field(const T& value) : value_(value) {}
       field(T&& value) : value_(std::move(value)) {}
       
       operator T&() { return value_; }
       operator const T&() const { return value_; }
       
       T& get() { return value_; }
       const T& get() const { return value_; }
       
       T* operator->() { return &value_; }
       const T* operator->() const { return &value_; }
       
   private:
       T value_;
   };
   
   } // namespace fory
   ```
   
   ### Alternative: Macro-based Approach
   
   For compatibility with existing codebases that can't change field types:
   
   ```cpp
   struct Foo {
       std::string f1;
       Bar f2;
       std::optional<std::string> f3;
       std::shared_ptr<Node> parent;
   };
   
   // Define field metadata separately
   FORY_FIELD_INFO(Foo,
       FORY_FIELD(f1, id = 0),
       FORY_FIELD(f2, id = 1),
       FORY_FIELD(f3, id = 2, nullable = true),
       FORY_FIELD(parent, id = 3, ref = true, nullable = true)
   );
   ```
   
   ### Design Decision: Required `id`
   
   The `id` template parameter is **required**:
   - `fory::id<0>` to `fory::id<N>`: Use tag ID encoding
   - `fory::id<-1>`: Explicit opt-out, use field name encoding
   
   Rationale:
   1. **Explicit control**: Using `fory::field<>` means opting into explicit control
   2. **Compile-time validation**: Template can static_assert uniqueness
   3. **Proven pattern**: Similar to protobuf field numbers
   
   ### Optimization Details
   
   #### 1. Non-nullable (Default) Optimization
   
   When `nullable` tag is NOT present:
   - Skip writing the null flag entirely (1 byte saved per field)
   - Directly serialize the field value
   - For `std::optional<T>`, must add `nullable` tag
   
   #### 2. No Ref Tracking (Default) Optimization
   
   When `ref` tag is NOT present:
   - Skip reference tracking map operations
   - Skip ref flag when combined with non-nullable
   - For `std::shared_ptr<T>`, consider adding `ref` tag if circular refs are possible
   
   #### 3. Tag ID Optimization
   
   When `id<N>` where N >= 0:
   - Field name encoded as varint instead of meta string
   - Significant space savings for long field names
   
   **Space savings:**
   
   | Field Name | Meta String (approx) | Tag ID |
   |------------|---------------------|--------|
   | `f1` | ~2 bytes | 1 byte |
   | `user_name` | ~6 bytes | 1 byte |
   | `transaction_id` | ~10 bytes | 1 byte |
   
   ### Implementation Notes
   
   1. **Template Metaprogramming**:
      - Use variadic templates to extract properties
      - Provide `constexpr` accessors for compile-time queries
      - Enable optimizations via `if constexpr`
   
   2. **Serializer Integration**:
      ```cpp
      template<typename T, typename... Props>
      struct Serializer<fory::field<T, Props...>> {
          static void write(Writer& writer, const fory::field<T, Props...>& f) {
              if constexpr (!fory::field<T, Props...>::is_nullable) {
                  // Skip null check, directly serialize
                  Serializer<T>::write(writer, f.get());
              } else {
                  // Write null flag, then value if not null
                  // ...
              }
          }
      };
      ```
   
   3. **Zero Overhead**:
      - `fory::field<T, ...>` should have same memory layout as `T`
      - All metadata is compile-time only
      - No runtime overhead compared to raw field
   
   4. **Validation**:
      - `static_assert` for duplicate tag IDs at compile time
      - `static_assert` for `id < -1`
      - Runtime error if non-nullable field has null value
   
   ### Performance Impact
   
   For a struct with 10 fields using default settings (non-nullable, no ref tracking):
   - **Space savings**: ~20 bytes per object (null + ref flags)
   - **CPU savings**: 10 fewer hash map operations per serialization
   - **Zero runtime overhead** for metadata (all compile-time)
   
   ## Additional context
   
   This is the C++ equivalent of Java's `@ForyField` annotation. See [Java issue #3000](https://github.com/apache/fory/issues/3000) for the original design discussion.
   
   Protocol spec: https://fory.apache.org/docs/specification/fory_xlang_serialization_spec


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org

Re: [I] [C++] Create fory::field<> template for field metadata [fory]

["chaokunyang (via GitHub)" <gi...@apache.org>]

chaokunyang commented on issue #3003:
URL: https://github.com/apache/fory/issues/3003#issuecomment-3690300932

   ## Implementation Progress
   
   The `fory::field<>` template has been implemented with a refined API design based on discussions. Here's a summary:
   
   ### Implemented Features
   
   #### 1. Core API (`cpp/fory/meta/field.h`)
   
   ```cpp
   namespace fory {
     // Tag types
     struct nullable {};  // Mark shared_ptr/unique_ptr as nullable
     struct not_null {};  // Explicit non-null (for future pointer types like weak_ptr)
     struct ref {};       // Enable reference tracking (shared_ptr only)
   
     template <typename T, int16_t Id, typename... Options>
     class field { /* ... */ };
   }
   ```
   
   #### 2. Design Decisions
   
   | Type | Default Nullable | Default Ref | Allowed Options |
   |------|-----------------|-------------|-----------------|
   | Primitives, `std::string` | **false** | false | **None** |
   | `std::optional<T>` | **true** (inherent) | false | **None** |
   | `std::shared_ptr<T>` | **false** | false | `nullable`, `not_null`, `ref` |
   | `std::unique_ptr<T>` | **false** | false | `nullable` only |
   
   #### 3. Compile-Time Validation
   
   ```cpp
   // ✅ Valid usage
   fory::field<std::string, 0> name;
   fory::field<std::optional<int32_t>, 1> optional_age;
   fory::field<std::shared_ptr<Node>, 2, fory::ref> parent;
   fory::field<std::shared_ptr<Node>, 3, fory::nullable, fory::ref> link;
   fory::field<std::unique_ptr<Data>, 4, fory::nullable> data;
   
   // ❌ Compile errors
   fory::field<int32_t, 0, fory::nullable> age;           // nullable on primitive
   fory::field<std::unique_ptr<T>, 1, fory::ref> ptr;     // ref on unique_ptr
   fory::field<std::optional<T>, 2, fory::nullable> opt;  // nullable on optional
   ```
   
   #### 4. Usage Example
   
   ```cpp
   struct Document {
     fory::field<std::string, 0> title;
     fory::field<int32_t, 1> version;
     fory::field<std::optional<std::string>, 2> description;
     fory::field<std::shared_ptr<User>, 3> author;                        // non-nullable
     fory::field<std::shared_ptr<User>, 4, fory::nullable> reviewer;      // nullable
     fory::field<std::shared_ptr<Node>, 5, fory::ref> parent;             // ref tracking
     fory::field<std::shared_ptr<Node>, 6, fory::nullable, fory::ref> p;  // both
   };
   
   FORY_STRUCT(Document, title, version, description, author, reviewer, parent, p);
   ```
   
   ### Files Created/Modified
   
   #### New Files
   - `cpp/fory/meta/field.h` - Core field template and type traits
   - `cpp/fory/meta/field_test.cc` - 22 unit tests for field metadata
   - `cpp/fory/serialization/field_serializer_test.cc` - 20 serialization round-trip tests
   
   #### Modified Files
   - `cpp/fory/meta/BUILD` - Added `field_test` target
   - `cpp/fory/meta/CMakeLists.txt` - Added `field.h` and test
   - `cpp/fory/serialization/BUILD` - Added `field_serializer_test` target
   - `cpp/fory/serialization/CMakeLists.txt` - Added test
   - `cpp/fory/serialization/struct_serializer.h` - Integration with field metadata
   - `cpp/fory/serialization/type_resolver.h` - Field type unwrapping for registration
   
   ### Test Results
   
   All tests pass:
   - 22 meta/field tests (type traits, field options, struct usage)
   - 20 serialization tests (primitives, optional, shared_ptr, unique_ptr, ref tracking, vectors)
   
   ### Deferred to Follow-up PR
   
   The `FORY_FIELD_TAGS` macro (non-invasive API for existing structs) was deferred due to complexity. The current implementation focuses on the `fory::field<>` wrapper approach.
   
   ### Branch
   
   Implementation is on branch `add_cpp_field_support`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org

Re: [I] [C++] Field metadata for optimized serialization [fory]

["chaokunyang (via GitHub)" <gi...@apache.org>]

chaokunyang closed issue #3003: [C++] Field metadata for optimized serialization
URL: https://github.com/apache/fory/issues/3003


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org