You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fory.apache.org by "pandalee99 (via GitHub)" <gi...@apache.org> on 2025/09/09 14:32:31 UTC
[I] [Roadmap] Full support for fory-cpp is planned in the future [fory]
pandalee99 opened a new issue, #2595:
URL: https://github.com/apache/fory/issues/2595
### Feature Request
I find that the community is developing very vigorously. Almost every language has its corresponding PR being generated, but the cpp seems not to have been updated for a long time
### Is your feature request related to a problem? Please describe
So I plan and promote the development of the cpp module
### Describe the solution you'd like
- I hope to upgrade the build tool of the cpp module first and upgrade bazel
- I will gradually implement some cross-language logic of the cpp module
- I hope that pyfory can use nanobind to share the calls of the cpp library, which means that the underlying code of the two modules is shared instead of using cython
- I hope to further enhance the execution efficiency of pyfory, such as simd and so on
- I hope to integrate fory into more modules for use in areas such as reasoning, distributed computing, and so on
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [I] [Roadmap] Full support for fory-cpp is planned in the future [fory]
Posted by "pandalee99 (via GitHub)" <gi...@apache.org>.
pandalee99 commented on issue #2595:
URL: https://github.com/apache/fory/issues/2595#issuecomment-3271333382
> Some things that came in my head, which needs be done:
>
> * [ ] Make arrow integration optional, not all users needs this integration[ ] Make fory cpp reflection system decoupled from arrow type system, [@PragmaTwice](https://github.com/PragmaTwice) added the first version of cpp reflection implementation, I'm not sure whether he still has time for this.[ ] Add cmake build support, since most cpp users use cmake for build[ ] Add type resolver for object graph in c++[ ] Add reference resolver for object graph in c++[ ] Build a macro/template based compile-codegen system for fory-cpp. Note that fory cpp use c++ 17, and we won't use c++ 20. Build compile-reflection systemm may take some efforts.[ ] Support serialization for basic types[ ] Support serialization for nested collection/map/set types using compile-codegen module[ ] Support serialization for struct and nested struct using compile-codegen module[ ] Support type meta encoding for schema evolution[ ] Support schema evolution serialization based on type meta, we need
to generate two version of deserializaion code at compile-time[ ] Support circular and shared reference such as shared_ptr, weak_ptr, unique_ptr
Yes, these are all very good ideas!
All these are functions that need to be gradually implemented in the future. However, I have another idea. In fact, pyfory can share a code logic with the cpp module and then call it using methods such as nanobind. This can further upgrade the old cython system and update the API at the same time.
Because I think very few people would directly call the c++ serializer. Such users seem not to be too many.
However, pyfory calls the cpp module at the bottom layer. I'm not sure if this compatibility is sufficient?
And pyfory still has greater potential for development, such as being integrated into ray? Or it can be used in various scientific computations, which are all very good ideas!
Also, what kind of thinking would the fory cpp reflection system have?
Will fory, in the future, follow arrow's example and implement inter-memory sharing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [I] [Roadmap] Full support for fory-cpp is planned in the future [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #2595:
URL: https://github.com/apache/fory/issues/2595#issuecomment-3271385989
@pandalee99 C++ don't have runtime reflection system. For serialization in c++, you msut generate coode at compile-time. The reflection system should do that. For c++ 17, you can take a search, it doesn't have reflection support. C++ 20 does have. But we need to keep c++17 for a broader users targets. And to implement reflection in c++17, you need to combine macro and templlate meta programming. @PragmaTwice has did some work in @1212. You can take a look at it if you are interested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [I] [Roadmap] Full support for fory-cpp is planned in the future [fory]
Posted by "pandalee99 (via GitHub)" <gi...@apache.org>.
pandalee99 commented on issue #2595:
URL: https://github.com/apache/fory/issues/2595#issuecomment-3271067960
If anyone in the community has ideas or would like to offer support, feel free to join the discussion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [I] [Roadmap] Full support for fory-cpp is planned in the future [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #2595:
URL: https://github.com/apache/fory/issues/2595#issuecomment-3271442019
For other questions:
1. inter-memory sharing is not our target, I don't think it's the goal of a serialization framework. It's more like some feature in an obejct store. And arrow does't support share memory, it's just buffers. It's the plasma that supports shared memory, but plasma has been deperacated. Or users allocate a buffer with share memory and write arrow data into it. In such cases, the share memory is done by users instead of by arrow. And the object graph format can't be used for random access, we have compression in the format. Even we have some kind of share memory, it doesn't provide any gains.
3. being integrated into other systems is not a development requirement
4. pyfory already call into cpp in the current implemention, the `_serialization.pyx` already used many c++ utils.
5. As for nanobind, maybe we need to make of POC and do some benchmark first. Ideally it will be faster and give more chanegs to share codebase between c++ and python implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [I] [Roadmap] Full support for fory-cpp [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang closed issue #2595: [Roadmap] Full support for fory-cpp
URL: https://github.com/apache/fory/issues/2595
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [I] [Roadmap] Full support for fory-cpp is planned in the future [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #2595:
URL: https://github.com/apache/fory/issues/2595#issuecomment-3271250802
Some things needs be done:
- [ ] Make arrow integration optional, not all users needs this integration
- [ ] Make fory cpp reflection system decoupled from arrow type system
- [ ] Add cmake build support, since most cpp users use cmake for build
- [ ] Add type resolver for object graph in c++
- [ ] Add reference resolver for object graph in c++
- [ ] Build a macro/template based compile-codegen system for fory-cpp. Note that fory cpp use c++ 17, and we won't use c++ 20. Build compile-reflection systemm may take some efforts.
- [ ] Support serialization for basic types
- [ ] Support serialization for nested collection/map/set types using compile-codegen module
- [ ] Support serialization for struct and nested struct using compile-codegen module
- [ ] Support type meta encoding for schema evolution
- [ ] Support schema evolution serialization based on type meta, we need to generate two version of deserializaion code at compile-time
- [ ] Support circular and shared reference such as shared_ptr, weak_ptr, unique_ptr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [I] [Roadmap] Full support for fory-cpp [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #2595:
URL: https://github.com/apache/fory/issues/2595#issuecomment-3578962412
finished in #2908
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org