You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fory.apache.org by "ruoruoniao (via GitHub)" <gi...@apache.org> on 2026/06/11 03:03:29 UTC
[I] [C++] Type is var but data is fixed when write `uint32_t`/`uint64_t`. [fory]
ruoruoniao opened a new issue, #3746:
URL: https://github.com/apache/fory/issues/3746
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/fory/issues) and found no similar issues.
### Version
Latest
### Component(s)
C++
### Minimal reproduce step
When use a normal struct with `uint32_t`, the type id is `0x0C`, but write data as 8 bytes like `0x01 0x00 0x00 0x00`.
### What did you expect to see?
If use `0x0C`, the data should be `0x01`.
### What did you see instead?
I see a comment at `Serializer<uint32_t>` and `Serializer<uint64_t>`:
```cpp
/// uint32_t serializer - uses VAR_UINT32 to match Rust xlang mode
template <> struct Serializer<uint32_t> {
... codes
/// uint64_t serializer - uses VAR_UINT64 to match Rust xlang mode
template <> struct Serializer<uint64_t> {
... codes
```
But if using a `struct`, in `struct_serializer.h -> write_struct_fields_impl` function, it will walk in branch:
```cpp
if constexpr (prim_count == total_count) {
...codes
}
```
So it will use `write_primitive_fields_fast` function to write data. It will let `uint32_t` field in `fixed_count`, so it will walk into branch:
```cpp
if constexpr (fixed_count > 0 && fixed_bytes > 0) {
write_fixed_primitive_fields<T>(obj, buffer,
std::make_index_sequence<fixed_count>{});
}
```
So maybe let `uint32_t` as a `varint` when use `FORY_STRUCT`?
### Anything Else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [I] [C++] Type is var but data is fixed when write `uint32_t`/`uint64_t`. [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang closed issue #3746: [C++] Type is var but data is fixed when write `uint32_t`/`uint64_t`.
URL: https://github.com/apache/fory/issues/3746
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [I] [C++] Type is var but data is fixed when write `uint32_t`/`uint64_t`. [fory]
Posted by "ruoruoniao (via GitHub)" <gi...@apache.org>.
ruoruoniao commented on issue #3746:
URL: https://github.com/apache/fory/issues/3746#issuecomment-4676833767
```cpp
template <typename FieldType, typename StructT, size_t Index>
constexpr bool configurable_int_is_fixed() {
if constexpr (is_signed_configurable_int_v<FieldType>) {
return field_int_encoding<FieldType, StructT, Index>() == Encoding::Fixed;
} else if constexpr (is_unsigned_configurable_int_v<FieldType>) {
return false;
} else {
return false;
}
}
template <typename FieldType, typename StructT, size_t Index>
constexpr bool configurable_int_is_varint() {
if constexpr (is_signed_configurable_int_v<FieldType>) {
return field_int_encoding<FieldType, StructT, Index>() != Encoding::Fixed;
} else if constexpr (is_unsigned_configurable_int_v<FieldType>) {
return true;
} else {
return false;
}
}
```
Maybe like this? ...I dont know...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [I] [C++] Type is var but data is fixed when write `uint32_t`/`uint64_t`. [fory]
Posted by "ruoruoniao (via GitHub)" <gi...@apache.org>.
ruoruoniao commented on issue #3746:
URL: https://github.com/apache/fory/issues/3746#issuecomment-4677685002
In serializer, vector<uint32_t> will copy data into it. But in python, there is all varint data. So it will went wrong at:
```cpp
// unsigne+_serializer.h
static inline std::vector<uint32_t> read_data(ReadContext &ctx) {
uint32_t total_bytes = ctx.read_var_uint32(ctx.error());
if (FORY_PREDICT_FALSE(ctx.has_error())) {
return std::vector<uint32_t>();
}
if (FORY_PREDICT_FALSE(total_bytes > ctx.config().max_binary_size)) {
ctx.set_error(Error::invalid_data("Binary size exceeds max_binary_size"));
return std::vector<uint32_t>();
}
// HERE!
if (total_bytes % sizeof(uint32_t) != 0) {
ctx.set_error(Error::invalid_data("Invalid length: " +
std::to_string(total_bytes)));
return std::vector<uint32_t>();
}
if (FORY_PREDICT_FALSE(total_bytes > ctx.buffer().remaining_size())) {
ctx.set_error(Error::invalid_data("Invalid length: " +
std::to_string(total_bytes)));
return std::vector<uint32_t>();
}
size_t length = total_bytes / sizeof(uint32_t);
std::vector<uint32_t> vec(length);
if (total_bytes > 0) {
ctx.read_bytes(vec.data(), total_bytes, ctx.error());
}
return vec;
}
```
Because data is varint, so...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [I] [C++] Type is var but data is fixed when write `uint32_t`/`uint64_t`. [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #3746:
URL: https://github.com/apache/fory/issues/3746#issuecomment-4676918020
If the int32 field is fixed int32, it will go to write_fixed_primitive_fields, otherwise, it wont' go to this function, the field value will still be writtten as varint32
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [I] [C++] Type is var but data is fixed when write `uint32_t`/`uint64_t`. [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #3746:
URL: https://github.com/apache/fory/issues/3746#issuecomment-4688007842
This is indeed a bug for unsigned int, let me fix it and release a new version.
Thanks for reporting this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [I] [C++] Type is var but data is fixed when write `uint32_t`/`uint64_t`. [fory]
Posted by "ruoruoniao (via GitHub)" <gi...@apache.org>.
ruoruoniao commented on issue #3746:
URL: https://github.com/apache/fory/issues/3746#issuecomment-4677643137
> If the int32 field is fixed int32, it will go to write_fixed_primitive_fields, otherwise, it wont' go to this function, the field value will still be writtten as varint32
No... If it works itself, all is OK, but if in `xlang`...
```cpp
//
// Created by MikuSoft on 2026/6/11.
//
#include <fory/serialization/fory.h>
#include <fory/serialization/serializer.h>
#include <fstream>
struct X {
uint32_t a = 0;
uint32_t b = 0;
uint32_t c = 0;
std::vector<uint32_t> d{};
};
FORY_STRUCT(X, a, b, c, d);
int main(const int argc, char **argv) {
fory::serialization::Fory fory = fory::serialization::ForyBuilder().xlang(true).track_ref(true).build();
fory.register_struct<X>("X");
const auto x = fory.serialize(X{66051, 66051, 66051, {66051, 66051, 66051}});
if (!x.ok()) {
return -1;
}
std::ofstream file("cxx.bin", std::ios::binary);
file.write(reinterpret_cast<const char *>(x.value().data()), static_cast<uint32_t>(x.value().size()));
file.close();
return 0;
}
```
```python
import pyfory
from pyfory.annotation import *
from dataclasses import dataclass, field
from typing import List
@dataclass
class X:
a: UInt32 = 0
b: UInt32 = 0
c: UInt32 = 0
d: List[UInt32] = field(default_factory=list)
"""
This will cause the same error.
@dataclass
class X:
a: FixedUInt32 = 0
b: FixedUInt32 = 0
c: FixedUInt32 = 0
d: List[FixedUInt32] = field(default_factory=list)
"""
fory = pyfory.Fory(xlang=True, ref=True, strict=True)
fory.register_type(X, typename="X")
with open("cxx.bin", "rb") as f:
data = fory.deserialize(f.read())
print(data)
with open("py.bin", "wb") as f:
f.write(fory.serialize(data))
```
Python will print `X(a=3, b=2, c=1, d=[])`. As you know, `66051` is `0x01 0x02 0x03`. Vector cannot be serialized correct. Also, it cannot convert into c++ by content in `py.bin`.
You can try yourself.
<img width="460" height="227" alt="Image" src="https://github.com/user-attachments/assets/77f01c37-8b6a-464a-815d-4918b1a52f55" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [I] [C++] Type is var but data is fixed when write `uint32_t`/`uint64_t`. [fory]
Posted by "ruoruoniao (via GitHub)" <gi...@apache.org>.
ruoruoniao commented on issue #3746:
URL: https://github.com/apache/fory/issues/3746#issuecomment-4677750103
Or use `0x0B` to write fix int. Or use `0x0C` to write var int.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org