You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "ollemartensson (via GitHub)" <gi...@apache.org> on 2025/08/31 21:59:09 UTC
[PR] C Data Interface [arrow-julia]
ollemartensson opened a new pull request, #561:
URL: https://github.com/apache/arrow-julia/pull/561
# Implement Apache Arrow C Data Interface for Zero-Copy Interoperability
## Overview
This PR implements the Apache Arrow C Data Interface specification to enable zero-copy data sharing between
Arrow.jl and other Arrow ecosystem implementations (PyArrow, Arrow C++, Rust, etc.).
## Research Foundation
This implementation is based on original research into:
- Apache Arrow C Data Interface ABI specification compliance requirements
- Memory management strategies for safe cross-language data sharing in Julia
- Zero-copy pointer passing mechanisms between Julia and foreign Arrow implementations
- Format string protocol optimization for Arrow type system interoperability
- Release callback patterns ensuring safe foreign memory lifecycle management
## Key Features
- **Full ABI Compatibility**: C-compatible structs (`CArrowSchema`, `CArrowArray`) with exact memory layout
matching Arrow specification
- **Comprehensive Type Support**: Format string encoding/decoding for all Arrow logical and physical types
- **Memory Safety**: GuardianObject system preventing premature GC, ImportedArrayHandle for foreign memory
management
- **Zero-Copy Performance**: Sub-microsecond pointer passing overhead with automatic cleanup
- **Robust Testing**: 37 comprehensive tests covering producer/consumer patterns and edge cases
## Technical Implementation
- Follows Apache Arrow C Data Interface v1.0 specification exactly
- Implements producer/consumer pattern with proper release callback handling
- Provides `export_to_c()` and `import_from_c()` functions for seamless interoperability
- Maintains Julia object lifecycles during foreign data sharing
## Testing
All tests pass independently on this branch. The implementation has been verified for:
- ✅ ABI compatibility with Arrow C specification
- ✅ Memory safety across GC cycles
- ✅ Type system round-trip fidelity
- ✅ Error handling for malformed inputs
## Development Methodology
Research and technical design conducted as original work. Implementation developed with AI assistance (Claude)
under direct technical guidance, following Apache Arrow specifications and established memory management
patterns.
**Ready for review and testing with other Arrow ecosystem tools.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] C Data Interface [arrow-julia]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #561:
URL: https://github.com/apache/arrow-julia/pull/561#issuecomment-3240487415
## [Codecov](https://app.codecov.io/gh/apache/arrow-julia/pull/561?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `362 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 4.48%. Comparing base ([`3712291`](https://app.codecov.io/gh/apache/arrow-julia/commit/37122911c24f44318e6d4a0840408adb3364cf2a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`043ac4a`](https://app.codecov.io/gh/apache/arrow-julia/commit/043ac4a3840fa483a79f7427228add3f402391dd?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 33 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/arrow-julia/pull/561?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [src/cdata/export.jl](https://app.codecov.io/gh/apache/arrow-julia/pull/561?src=pr&el=tree&filepath=src%2Fcdata%2Fexport.jl&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL2NkYXRhL2V4cG9ydC5qbA==) | 0.00% | [134 Missing :warning: ](https://app.codecov.io/gh/apache/arrow-julia/pull/561?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [src/cdata/import.jl](https://app.codecov.io/gh/apache/arrow-julia/pull/561?src=pr&el=tree&filepath=src%2Fcdata%2Fimport.jl&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL2NkYXRhL2ltcG9ydC5qbA==) | 0.00% | [129 Missing :warning: ](https://app.codecov.io/gh/apache/arrow-julia/pull/561?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [src/cdata/format.jl](https://app.codecov.io/gh/apache/arrow-julia/pull/561?src=pr&el=tree&filepath=src%2Fcdata%2Fformat.jl&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL2NkYXRhL2Zvcm1hdC5qbA==) | 0.00% | [83 Missing :warning: ](https://app.codecov.io/gh/apache/arrow-julia/pull/561?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [src/cdata/structs.jl](https://app.codecov.io/gh/apache/arrow-julia/pull/561?src=pr&el=tree&filepath=src%2Fcdata%2Fstructs.jl&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL2NkYXRhL3N0cnVjdHMuamw=) | 0.00% | [16 Missing :warning: ](https://app.codecov.io/gh/apache/arrow-julia/pull/561?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (3712291) and HEAD (043ac4a). Click for more details.
>
> <details><summary>HEAD has 28 uploads less than BASE</summary>
>
>| Flag | BASE (3712291) | HEAD (043ac4a) |
>|------|------|------|
>||35|7|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #561 +/- ##
==========================================
- Coverage 87.43% 4.48% -82.96%
==========================================
Files 26 30 +4
Lines 3288 3681 +393
==========================================
- Hits 2875 165 -2710
- Misses 413 3516 +3103
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/arrow-julia/pull/561?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] C Data Interface [arrow-julia]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #561:
URL: https://github.com/apache/arrow-julia/pull/561#discussion_r2317529465
##########
examples/cdata_demo.jl:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
Review Comment:
Could you re-copy this part from other file?
Folding is different.
##########
examples/cdata_demo.jl:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
Review Comment:
Could you use the exactly same header text as other files?
```suggestion
# http://www.apache.org/licenses/LICENSE-2.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] C Data Interface [arrow-julia]
Posted by "vustef (via GitHub)" <gi...@apache.org>.
vustef commented on PR #561:
URL: https://github.com/apache/arrow-julia/pull/561#issuecomment-3541855166
@kou @ollemartensson thanks for starting this, this would be an awesome addition. Curious though, do you plan to continue pushing this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] C Data Interface [arrow-julia]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #561:
URL: https://github.com/apache/arrow-julia/pull/561#issuecomment-3247328860
Could you add `Generated-by: ` trailer to PR description (that will be used as the commit message on merge)?
It's recommended in https://www.apache.org/legal/generative-tooling.html .
We should follow the guidance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org