[I] Unknown flags should be zeroed when deep copying [arrow-nanoarrow]

["bkietz (via GitHub)" <gi...@apache.org>]

bkietz opened a new issue, #312:
URL: https://github.com/apache/arrow-nanoarrow/issues/312

   The C data interface doesn't define deep copying of arrays, so a deep copy is logically an import followed by an export. New but backward compatible flags could be added which are unknown to nanoarrow on import. Since nanoarrow does not know how to uphold the guarantees implied by an unknown flag when exporting, those unknown flags should probably be zeroed in the copy instead of being passed through unmodified.
   
   As a trivial example, the flag might indicate that the buffers are allocated in a particular memory space (like cuda or hip's `unified` memory). A deep copy's buffers would be allocated elsewhere, and passing through the unified memory flag could break a consumer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Unknown flags should be zeroed when deep copying [arrow-nanoarrow]

["paleolimbot (via GitHub)" <gi...@apache.org>]

paleolimbot commented on issue #312:
URL: https://github.com/apache/arrow-nanoarrow/issues/312#issuecomment-1785942041

   It may help to think of nanoarrow as a C Data interface helper toolkit than an Arrow implementation. There is no Import in nanoarrow...there are just `ArrowSchema`s, `ArrowArray`s, and `ArrowArrayStream`s and things you might want to know about them. There are also some physical manipulation tools, which is where deep copying comes in.
   
   An example of a "deep copy" operation where modifying anything about the output would be questionable or maybe dangerous or maybe annoying is the "basic array stream". It takes an `ArrowSchema`, which it moves from the caller and owns, and zero or more `ArrowArray`s, which are also moved from the caller and owned until the `get_next` callback is called for that particular array's position in the queue. The `get_schema` callback has to perform a deep copy ( https://github.com/apache/arrow-nanoarrow/blob/main/src/nanoarrow/array_stream.c#L29-L38 ) because it needs to be able to be called more than once and because the spec states that the caller is responsible for managing the lifecycle of the returned schema. There's an independent implementation of that in ADBC, too, that uses nanoarrow's deep copy. There are places that a caller of `ArrowSchemaDeepCopy()` should perhaps check the `flags` values, but it's not an inherent problem of the deep copy operation.
   
   There are some other places that could benefit from checking flags, like `ArrowArrayViewInitFromSchema()` and perhaps `ArrowArrayInitFromSchema()`.
   
   > As a trivial example, the flag might indicate that the buffers are allocated in a particular memory space
   
   It would be worth clarifying in the specification whether this kind of thing will ever be one of the uses of an `ArrowSchema`. Right now they are a somewhat direct translation of an `arrow::Field`. It would be very strange to store information about where an `arrow::Array`'s buffers lived in the corresponding `arrow::Field`, and so I don't think anybody expects them to be used in this way at the moment. 
   
   Arrow opted to handle address spaces using the device array instead of schema metadata/flags, which was kind of another implicit vote towards the `ArrowSchema` as an independent data type/field concept not tied to a specific `ArrowArray`. I don't think it's impossible to update that precedent, nor do I think it would actually cause any problems if done carefully, but it's also a reasonably large conceptual update that shouldn't be taken lightly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Unknown flags should be zeroed when deep copying [arrow-nanoarrow]

["paleolimbot (via GitHub)" <gi...@apache.org>]

paleolimbot closed issue #312: Unknown flags should be zeroed when deep copying
URL: https://github.com/apache/arrow-nanoarrow/issues/312


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org