You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Virgiel (via GitHub)" <gi...@apache.org> on 2023/06/23 13:25:26 UTC
[GitHub] [arrow-rs] Virgiel opened a new pull request, #4447: Simplify ffi import/export
Virgiel opened a new pull request, #4447:
URL: https://github.com/apache/arrow-rs/pull/4447
# Which issue does this PR close?
Closes #4444
# Rationale for this change
This simplification remove some redundant code and simplify the ffi logic around towo functions `from_ffi` and `to_ffi`.
# What changes are included in this PR?
TODO
# Are there any user-facing changes?
TODO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a diff in pull request #4447: Simplify ffi import/export
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #4447:
URL: https://github.com/apache/arrow-rs/pull/4447#discussion_r1242592070
##########
arrow/src/ffi.rs:
##########
@@ -222,41 +221,69 @@ unsafe fn create_buffer(
.map(|ptr| Buffer::from_custom_allocation(ptr, len, owner))
}
-pub trait ArrowArrayRef {
- fn to_data(&self) -> Result<ArrayData> {
- let data_type = self.data_type()?;
- let len = self.array().len();
- let offset = self.array().offset();
- let null_count = self.array().null_count();
+/// Export to the C Data Interface
+pub fn to_ffi(data: &ArrayData) -> Result<(FFI_ArrowArray, FFI_ArrowSchema)> {
+ let array = FFI_ArrowArray::new(data);
+ let schema = FFI_ArrowSchema::try_from(data.data_type())?;
+ Ok((array, schema))
+}
+
+/// Import [ArrayData] from the C Data Interface
+///
+/// # Safety
+///
+/// This struct assumes that the incoming data agrees with the C data interface.
+pub fn from_ffi(array: FFI_ArrowArray, schema: &FFI_ArrowSchema) -> Result<ArrayData> {
+ let array = Arc::new(array);
+ let tmp = Cursor {
+ array: &array,
+ schema,
+ owner: &array,
+ };
+ tmp.consume()
+}
+
+#[derive(Debug)]
+struct Cursor<'a> {
Review Comment:
This is not going to move, not sure why naming it as Cursor. Personally preferring original name ArrowArray. As this is internal struct, not a big deal though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] Virgiel commented on a diff in pull request #4447: Simplify ffi import/export
Posted by "Virgiel (via GitHub)" <gi...@apache.org>.
Virgiel commented on code in PR #4447:
URL: https://github.com/apache/arrow-rs/pull/4447#discussion_r1242641951
##########
arrow/src/ffi.rs:
##########
@@ -222,41 +221,69 @@ unsafe fn create_buffer(
.map(|ptr| Buffer::from_custom_allocation(ptr, len, owner))
}
-pub trait ArrowArrayRef {
- fn to_data(&self) -> Result<ArrayData> {
- let data_type = self.data_type()?;
- let len = self.array().len();
- let offset = self.array().offset();
- let null_count = self.array().null_count();
+/// Export to the C Data Interface
+pub fn to_ffi(data: &ArrayData) -> Result<(FFI_ArrowArray, FFI_ArrowSchema)> {
+ let array = FFI_ArrowArray::new(data);
+ let schema = FFI_ArrowSchema::try_from(data.data_type())?;
+ Ok((array, schema))
+}
+
+/// Import [ArrayData] from the C Data Interface
+///
+/// # Safety
+///
+/// This struct assumes that the incoming data agrees with the C data interface.
+pub fn from_ffi(array: FFI_ArrowArray, schema: &FFI_ArrowSchema) -> Result<ArrayData> {
+ let array = Arc::new(array);
+ let tmp = Cursor {
+ array: &array,
+ schema,
+ owner: &array,
+ };
+ tmp.consume()
+}
+
+#[derive(Debug)]
+struct Cursor<'a> {
Review Comment:
It moves down to the owner's child array, but 'cursor' may be a little misleading, so it will move back to ArrowArray
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a diff in pull request #4447: Simplify ffi import/export
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #4447:
URL: https://github.com/apache/arrow-rs/pull/4447#discussion_r1242493617
##########
arrow/src/ffi.rs:
##########
@@ -222,41 +221,69 @@ unsafe fn create_buffer(
.map(|ptr| Buffer::from_custom_allocation(ptr, len, owner))
}
-pub trait ArrowArrayRef {
- fn to_data(&self) -> Result<ArrayData> {
- let data_type = self.data_type()?;
- let len = self.array().len();
- let offset = self.array().offset();
- let null_count = self.array().null_count();
+/// Export to the C Data Interface
+pub fn to_ffi(data: &ArrayData) -> Result<(FFI_ArrowArray, FFI_ArrowSchema)> {
+ let array = FFI_ArrowArray::new(data);
+ let schema = FFI_ArrowSchema::try_from(data.data_type())?;
+ Ok((array, schema))
+}
+
+/// Import [ArrayData] from the C Data Interface
+///
+/// # Safety
+///
+/// This struct assumes that the incoming data agrees with the C data interface.
+pub fn from_ffi(array: FFI_ArrowArray, schema: &FFI_ArrowSchema) -> Result<ArrayData> {
Review Comment:
Hmm, who is going to maintain `FFI_ArrowArray` ownership? Looks like in `Cursor` it only keeps reference to it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] Virgiel commented on a diff in pull request #4447: Simplify ffi import/export
Posted by "Virgiel (via GitHub)" <gi...@apache.org>.
Virgiel commented on code in PR #4447:
URL: https://github.com/apache/arrow-rs/pull/4447#discussion_r1242560260
##########
arrow/src/ffi.rs:
##########
@@ -222,41 +221,69 @@ unsafe fn create_buffer(
.map(|ptr| Buffer::from_custom_allocation(ptr, len, owner))
}
-pub trait ArrowArrayRef {
- fn to_data(&self) -> Result<ArrayData> {
- let data_type = self.data_type()?;
- let len = self.array().len();
- let offset = self.array().offset();
- let null_count = self.array().null_count();
+/// Export to the C Data Interface
+pub fn to_ffi(data: &ArrayData) -> Result<(FFI_ArrowArray, FFI_ArrowSchema)> {
+ let array = FFI_ArrowArray::new(data);
+ let schema = FFI_ArrowSchema::try_from(data.data_type())?;
+ Ok((array, schema))
+}
+
+/// Import [ArrayData] from the C Data Interface
+///
+/// # Safety
+///
+/// This struct assumes that the incoming data agrees with the C data interface.
+pub fn from_ffi(array: FFI_ArrowArray, schema: &FFI_ArrowSchema) -> Result<ArrayData> {
Review Comment:
The ownership is transferred to ArrayData in the same way as before. As I understand it, this happens when the arc is cloned in the buffer logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a diff in pull request #4447: Simplify ffi import/export
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #4447:
URL: https://github.com/apache/arrow-rs/pull/4447#discussion_r1242588995
##########
arrow/src/ffi.rs:
##########
@@ -222,41 +221,69 @@ unsafe fn create_buffer(
.map(|ptr| Buffer::from_custom_allocation(ptr, len, owner))
}
-pub trait ArrowArrayRef {
- fn to_data(&self) -> Result<ArrayData> {
- let data_type = self.data_type()?;
- let len = self.array().len();
- let offset = self.array().offset();
- let null_count = self.array().null_count();
+/// Export to the C Data Interface
+pub fn to_ffi(data: &ArrayData) -> Result<(FFI_ArrowArray, FFI_ArrowSchema)> {
+ let array = FFI_ArrowArray::new(data);
+ let schema = FFI_ArrowSchema::try_from(data.data_type())?;
+ Ok((array, schema))
+}
+
+/// Import [ArrayData] from the C Data Interface
+///
+/// # Safety
+///
+/// This struct assumes that the incoming data agrees with the C data interface.
+pub fn from_ffi(array: FFI_ArrowArray, schema: &FFI_ArrowSchema) -> Result<ArrayData> {
Review Comment:
Oh, I see. Cursor is just temporary struct which doesn't actually need to hold ownership like ArrowArray before.
The refactoring just directly consumes FFI_ArrowArray and FFI_ArrowSchema into ArrayData.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold merged pull request #4447: Simplify ffi import/export
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold merged PR #4447:
URL: https://github.com/apache/arrow-rs/pull/4447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org