You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "DrChainsaw (via GitHub)" <gi...@apache.org> on 2023/06/05 09:40:41 UTC
[GitHub] [arrow-julia] DrChainsaw opened a new issue, #449: Should extension metadata tag be more specific?
DrChainsaw opened a new issue, #449:
URL: https://github.com/apache/arrow-julia/issues/449
The current implementation seems to use the metadata tag `"ARROW:extension:name"` for julia specific types (e.g Symbols).
Shouldn't this instead be something like `"ARROW:extension:arrow-julia.Symbol"` to make collisions less likely? Maybe I misunderstand the [extension documentation](https://arrow.apache.org/docs/format/Columnar.html#custom-application-metadata), but to me it seems like the tag `name` (and maybe also `extension`, docs are not super clear imo) is an example placeholder to be replaced with the actual name.
With the above change it would be possible to write tables from other languages which make use of custom types in more than one language. For example, I'm writing tables from the java implementation and in many cases I would like strings to be deserialized as symbols (e.g. for enums), but that non-specific tag might block similar optimizations for other implementations.
I guess it is not fun to change it due to backwards compatibility (probably need to support both tags for the forseeable future), but maybe better to rip of the bandaid as quickly as possible should it turn out that it should use a more specific tag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] DrChainsaw commented on issue #449: Should extension metadata tag be more specific?
Posted by "DrChainsaw (via GitHub)" <gi...@apache.org>.
DrChainsaw commented on issue #449:
URL: https://github.com/apache/arrow-julia/issues/449#issuecomment-1577140314
After searching through the arrow repo after `arrow:extension` is seems like you might be right. [Here](https://github.com/apache/arrow/blob/f5a57292e5187b1a1640b04303e9f6b63135892a/cpp/src/arrow/extension_type.cc#L178) it is defined as a const in the cpp code for example and I could not find any trace of it being manipulated or changed anywhere (which would be a quite strange thing to do as well).
The [c code](https://github.com/apache/arrow/blob/f5a57292e5187b1a1640b04303e9f6b63135892a/cpp/src/arrow/c/bridge.cc#L483) seems to accept the metadata as a vector of pairs through, so it would in theory allow for multiple identical keys, but python, java and Julia use dicts so there is no way to have it through any of them.
It is a bit unclear to me what the ExtensionType stuff does in arrow and what one gets for buying in to it. However, most things points to it being a mechanism for your `foo` example.
I close the issue now since my initial understanding of the extension metadata was incorrect and I don't think there is any action needed here. Just reopen or open another issue if there is a point about late conversion stuff (e.g. String <-> Symbol) not fitting into the definition of extensions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] DrChainsaw closed issue #449: Should extension metadata tag be more specific?
Posted by "DrChainsaw (via GitHub)" <gi...@apache.org>.
DrChainsaw closed issue #449: Should extension metadata tag be more specific?
URL: https://github.com/apache/arrow-julia/issues/449
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] ericphanson commented on issue #449: Should extension metadata tag be more specific?
Posted by "ericphanson (via GitHub)" <gi...@apache.org>.
ericphanson commented on issue #449:
URL: https://github.com/apache/arrow-julia/issues/449#issuecomment-1576671366
Maybe we can check against another implementation by seeing the metadata produced by an extension type there, e.g. following https://arrow.apache.org/docs/python/generated/pyarrow.ExtensionType.html#pyarrow.ExtensionType.extension_name.
It doesn't seem clear if a column can have more than one extension type though. It could be there's only 1 key on purpose so that different implementations can share that key to define an extension to the arrow spec overall (e.g. we if we all agree what a `foo` is, we define an extension name for that, serialize that metadata, and then read it in as a `foo` when possible). Which maybe then means your suggestion is that arrow-julia shouldn't be using "extension types" specifically for metadata that is only used by that implementation, and should use other keys for that.
Would appreciate any feedback from someone who understands the spec better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] ericphanson commented on issue #449: Should extension metadata tag be more specific?
Posted by "ericphanson (via GitHub)" <gi...@apache.org>.
ericphanson commented on issue #449:
URL: https://github.com/apache/arrow-julia/issues/449#issuecomment-1576484339
Just below they suggest name spacing the value passed there: https://arrow.apache.org/docs/format/Columnar.html#extension-types. That bit reads to me like it is not a placeholder, but rather the customization is in the value (not the key).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] DrChainsaw commented on issue #449: Should extension metadata tag be more specific?
Posted by "DrChainsaw (via GitHub)" <gi...@apache.org>.
DrChainsaw commented on issue #449:
URL: https://github.com/apache/arrow-julia/issues/449#issuecomment-1576517046
But the metadata is a dict, so the namespacing they suggest would be pointless if only applied to values since they will be overwritten.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org