You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "yevgenypats (via GitHub)" <gi...@apache.org> on 2023/02/22 10:51:12 UTC
[GitHub] [arrow] yevgenypats opened a new issue, #34292: [GO] arrjson is internal
yevgenypats opened a new issue, #34292:
URL: https://github.com/apache/arrow/issues/34292
### Describe the bug, including details regarding any error messages, version, and platform.
Hey there! I was trying to use the arrow to json writer and seems like `github.com/apache/arrow/go/v12/arrow/internal/arrjson` is an internal package for some reason which makes it impossible to import from any other library. is it on purpose or a mistake? If a mistake I can open a PR moving it to non internal package.
### Component(s)
Go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] yevgenypats closed issue #34292: [GO] arrjson is internal
Posted by "yevgenypats (via GitHub)" <gi...@apache.org>.
yevgenypats closed issue #34292: [GO] arrjson is internal
URL: https://github.com/apache/arrow/issues/34292
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] yevgenypats closed issue #34292: [Go] arrjson is internal
Posted by "yevgenypats (via GitHub)" <gi...@apache.org>.
yevgenypats closed issue #34292: [Go] arrjson is internal
URL: https://github.com/apache/arrow/issues/34292
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #34292: [Go] arrjson is internal
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #34292:
URL: https://github.com/apache/arrow/issues/34292#issuecomment-1454201790
Just adding an `UnmarshalJSON` for the extension builder would work for just the extension builder, but wouldn't work for the nested scenarios. It might make sense to just make the `Unmarshal` method public on the builders which would then allow customizing when combined with adding the functionality to have it properly construct a custom builder type to wrap the ExtensionBuilder. Since all nested scenarios and `FromJSON` interact by way of calling the `unmarshal` function of the interface, making that function public would allow it to be overridden by custom builders.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] yevgenypats commented on issue #34292: [GO] arrjson is internal
Posted by "yevgenypats (via GitHub)" <gi...@apache.org>.
yevgenypats commented on issue #34292:
URL: https://github.com/apache/arrow/issues/34292#issuecomment-1445160783
Not sure, if it's the right issue but didn't want to open another.
Im playing with the default MarshalJSON but actually was having trouble implementing MarshalJSON for custom arrow extension Types. Any pointer? Or I would have to implementing something like the internal `arrjson` and if yes that would bring my point back if it is worth then making this an external package?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #34292: [Go] arrjson is internal
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #34292:
URL: https://github.com/apache/arrow/issues/34292#issuecomment-1447109734
> Is there a reason why getOneForMarshal is private in the first place (or just basically keeping everything private until public is needed which is a good reason and then we can make it public :) )
It's entirely keeping everything private until public is needed to reduce the public API and thus allow easier changes. :smile:
Making `getOneForMarshal` public is definitely an option. For now it was just an implementation detail for marshalling, but it makes sense to make it public in order to ease customization of the behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #34292: [Go] arrjson is internal
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #34292:
URL: https://github.com/apache/arrow/issues/34292#issuecomment-1448564484
I can think of a couple ways to go about this.
* Adding methods to the ExtensionType itself which can be used / define how to marshal/unmarshal the data itself (not just serializing and deserializing the type)
* Accepting an optional Unmarshaller object when creating an extension builder and then having the builder determine whether to delegate to it if it exists or to delegate to the underlying storage builder if it isn't set.
* Following the `ExtensionArray` method by making the ExtensionBuilder a "must-embed" type and an ExtensionBuilder interface that includes `UnmarshalJSON` along with adding a function to the extension type to return the type of the extension builder to use (like what is done for returning a `reflect.Type` for the ExtensionArray type)
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #34292: [Go] arrjson is internal
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #34292:
URL: https://github.com/apache/arrow/issues/34292#issuecomment-1446791648
Hmm, that's a good point. You can override the `MarshalJSON` method, but you're right. It won't cover the case for `RecordToJSON` or for marshalling to JSON from inside a nested type. Exposing the `arrjson` internal pkg wouldn't do anything to help that either. We should modify the `ExtensionArrayBase` struct so that the behavior can be more customizable. Possibly by allowing a way to set a custom JSON marshaller into it. If you think of a good way to implement it, I'll happily review the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] yevgenypats commented on issue #34292: [GO] arrjson is internal
Posted by "yevgenypats (via GitHub)" <gi...@apache.org>.
yevgenypats commented on issue #34292:
URL: https://github.com/apache/arrow/issues/34292#issuecomment-1440148261
Hey @zeroshade , that seems perfect. Thank you for the quick turn-around!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] yevgenypats commented on issue #34292: [Go] arrjson is internal
Posted by "yevgenypats (via GitHub)" <gi...@apache.org>.
yevgenypats commented on issue #34292:
URL: https://github.com/apache/arrow/issues/34292#issuecomment-1448398018
Seems like it's a deeper rabbit hole here for custom json marshaling/unmarshaling for extensions.
For example here - https://github.com/apache/arrow/blob/main/go/arrow/array/extension.go#L228
The extension [builder](https://github.com/apache/arrow/blob/main/go/arrow/array/extension.go#L228) is always calling the underlying storage builder which means we can't really control `unmarshal` for extensions.
Don't have a good idea on top of my head but seems like the extension abstraction might need a bit of more work or maybe we need a storage extension?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] yevgenypats commented on issue #34292: [Go] arrjson is internal
Posted by "yevgenypats (via GitHub)" <gi...@apache.org>.
yevgenypats commented on issue #34292:
URL: https://github.com/apache/arrow/issues/34292#issuecomment-1446653210
Maybe I missed something but it didn't work for me as it always went ahead and used the underlying storage `getOneForMarshal`.
For example [here](https://github.com/apache/arrow/blob/main/go/arrow/array/util.go#L231) it calls `getOneForMarshal` which cannot be implemented by an extension type (I think).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #34292: [Go] arrjson is internal
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #34292:
URL: https://github.com/apache/arrow/issues/34292#issuecomment-1446546736
What was the trouble you were having for implementing it for a custom arrow extension type?
In theory you should be able to just implement the MarshalJSON for the extension array type, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] yevgenypats commented on issue #34292: [Go] arrjson is internal
Posted by "yevgenypats (via GitHub)" <gi...@apache.org>.
yevgenypats commented on issue #34292:
URL: https://github.com/apache/arrow/issues/34292#issuecomment-1448695539
It seems reasonable but I guess I still miss something in the code flow, if you can direct me.
Let's take [FromJSON](https://github.com/apache/arrow/blob/main/go/arrow/array/util.go#L124)
This calls [bldr.unmarshal](https://github.com/apache/arrow/blob/main/go/arrow/array/util.go#L164) which just calls an internal `unmarshal` method that for the extension case just calls the underlying private [unmarshal](https://github.com/apache/arrow/blob/main/go/arrow/array/extension.go#L228) method which we don't have control on.
So the question is, do I need to change how `FromJSON` works, should we add more capabilities to every builder. This is where Im mostly stuck at the moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] yevgenypats commented on issue #34292: [Go] arrjson is internal
Posted by "yevgenypats (via GitHub)" <gi...@apache.org>.
yevgenypats commented on issue #34292:
URL: https://github.com/apache/arrow/issues/34292#issuecomment-1471776702
Closing as this was addressed here: https://github.com/apache/arrow/issues/34453
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] yevgenypats commented on issue #34292: [Go] arrjson is internal
Posted by "yevgenypats (via GitHub)" <gi...@apache.org>.
yevgenypats commented on issue #34292:
URL: https://github.com/apache/arrow/issues/34292#issuecomment-1447096059
> Hmm, that's a good point. You can override the `MarshalJSON` method, but you're right. It won't cover the case for `RecordToJSON` or for marshalling to JSON from inside a nested type. Exposing the `arrjson` internal pkg wouldn't do anything to help that either. We should modify the `ExtensionArrayBase` struct so that the behavior can be more customizable. Possibly by allowing a way to set a custom JSON marshaller into it. If you think of a good way to implement it, I'll happily review the PR.
Not sure but what about making `getOneForMarshal` public I think then it should be possible to overwrite it. Given Im new to the codebase, Is there a reason why `getOneForMarshal` is private in the first place (or just basically keeping everything private until public is needed which is a good reason and then we can make it public :) )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #34292: [GO] arrjson is internal
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #34292:
URL: https://github.com/apache/arrow/issues/34292#issuecomment-1440128425
Hi @yevgenypats,
`arrjson` is internal on purpose as it is explicitly the implementation for Arrow's Integration testing format as defined here: https://arrow.apache.org/docs/format/Integration.html
If you want generic JSON read/write for Arrow arrays, all Arrow Arrays have implemented the `MarshalJSON` interface and should be able to generically marshal any Arrow data to JSON for writing.
In addition there are several `FromJSON` methods in the `array` package, such as https://pkg.go.dev/github.com/apache/arrow/go/v11@v11.0.0/arrow/array#FromJSON
Is that sufficient?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org