[Go] expose ability to write arrow.Table to JSON

[Agam Brahma <ag...@gmail.com>]

Hi all,

I was wondering why `arrjson` is kept as an internal package within go/arrow.

Is there a different recommended way to serialize an instance of `arrow.Table` to JSON? afaict, using `arrjson.Writer` seems like a great way to do this, but it isn't possible to depend on it as an external user, because of it residing within the `.../internal/...` path.

Any pointers are much appreciated.

Thanks,
Agam

Re: RE: [Go] expose ability to write arrow.Table to JSON

[Francois Saint-Jacques <fs...@gmail.com>]

You can either use the provided server facility found in flight [1],
or use stream directly via ipc [2]. You can look at the tests on how
to use both facilities.

François

[1] https://github.com/apache/arrow/tree/master/go/arrow/flight
[2] https://github.com/apache/arrow/tree/master/go/arrow/ipc

On Thu, Apr 22, 2021 at 12:39 PM Agam Brahma <ag...@gmail.com> wrote:
>
> Thanks for the clarifications, much appreciated.
>
> Looking closer, I realize `arrjson` is anyway separating out the values, which isn't what I'd want to ship a table from one service to another.
>
> What's a good way to embed the table as a byte stream that can be "read back out" the other end?
>
> I see `arrio.Copy` being publicly callable, and a `TableReader` should serve as its reader, what would the `Writer` be in this case? The one in the `ipc` package ?
>
> On 2021/04/22 14:26:39, Matthew Topol <mt...@factset.com> wrote:
> > Micah is correct, the arrjson package is used for the internal integration testing using the specific JSON format for that integration testing which is not likely what Users would want when converting Arrow to JSON.
> >
> > There is not currently a recommended way to serialize an instance of arrow.Table to JSON because there is not standardized external format expected as JSON for arrow to my knowledge. Having arrjson be internal is intentional due to it being for the internal integration testing.
> >
> > -----Original Message-----
> > From: Micah Kornfield <em...@gmail.com>
> > Sent: Wednesday, April 21, 2021 11:19 PM
> > To: dev <de...@arrow.apache.org>
> > Subject: Re: [Go] expose ability to write arrow.Table to JSON
> >
> > > I was wondering why `arrjson` is kept as an internal package within
> > go/arrow.
> >
> > I think this is probably what is used for internal integration testing, we have specific JSON format that is expected, that needs to free to evolve and probably isn't what users are looking for.
> >
> >
> > > Is there a different recommended way to serialize an instance of
> > `arrow.Table` to JSON? afaict, using `arrjson.Writer` seems like a great way to do this, but it isn't possible to depend on it as an external user, because of it residing within the `.../internal/...` path.
> >
> > Sorry I don't know the answer to this, but I would guess there isn't anything currently in the Go package for this.
> >
> >
> > On Wed, Apr 21, 2021 at 6:45 PM Agam Brahma <ag...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > I was wondering why `arrjson` is kept as an internal package within
> > > go/arrow.
> > >
> > > Is there a different recommended way to serialize an instance of
> > > `arrow.Table` to JSON? afaict, using `arrjson.Writer` seems like a
> > > great way to do this, but it isn't possible to depend on it as an
> > > external user, because of it residing within the `.../internal/...` path.
> > >
> > > Any pointers are much appreciated.
> > >
> > > Thanks,
> > > Agam
> > >
> > >
> > >
> > >
> >

Re: RE: [Go] expose ability to write arrow.Table to JSON

[Agam Brahma <ag...@gmail.com>]

Thanks for the clarifications, much appreciated.

Looking closer, I realize `arrjson` is anyway separating out the values, which isn't what I'd want to ship a table from one service to another.

What's a good way to embed the table as a byte stream that can be "read back out" the other end?

I see `arrio.Copy` being publicly callable, and a `TableReader` should serve as its reader, what would the `Writer` be in this case? The one in the `ipc` package ?

On 2021/04/22 14:26:39, Matthew Topol <mt...@factset.com> wrote: 
> Micah is correct, the arrjson package is used for the internal integration testing using the specific JSON format for that integration testing which is not likely what Users would want when converting Arrow to JSON.
> 
> There is not currently a recommended way to serialize an instance of arrow.Table to JSON because there is not standardized external format expected as JSON for arrow to my knowledge. Having arrjson be internal is intentional due to it being for the internal integration testing.
> 
> -----Original Message-----
> From: Micah Kornfield <em...@gmail.com> 
> Sent: Wednesday, April 21, 2021 11:19 PM
> To: dev <de...@arrow.apache.org>
> Subject: Re: [Go] expose ability to write arrow.Table to JSON
> 
> > I was wondering why `arrjson` is kept as an internal package within
> go/arrow.
> 
> I think this is probably what is used for internal integration testing, we have specific JSON format that is expected, that needs to free to evolve and probably isn't what users are looking for.
> 
> 
> > Is there a different recommended way to serialize an instance of
> `arrow.Table` to JSON? afaict, using `arrjson.Writer` seems like a great way to do this, but it isn't possible to depend on it as an external user, because of it residing within the `.../internal/...` path.
> 
> Sorry I don't know the answer to this, but I would guess there isn't anything currently in the Go package for this.
> 
> 
> On Wed, Apr 21, 2021 at 6:45 PM Agam Brahma <ag...@gmail.com> wrote:
> 
> > Hi all,
> >
> > I was wondering why `arrjson` is kept as an internal package within 
> > go/arrow.
> >
> > Is there a different recommended way to serialize an instance of 
> > `arrow.Table` to JSON? afaict, using `arrjson.Writer` seems like a 
> > great way to do this, but it isn't possible to depend on it as an 
> > external user, because of it residing within the `.../internal/...` path.
> >
> > Any pointers are much appreciated.
> >
> > Thanks,
> > Agam
> >
> >
> >
> >
> 

RE: [Go] expose ability to write arrow.Table to JSON

[Matthew Topol <mt...@factset.com>]

Micah is correct, the arrjson package is used for the internal integration testing using the specific JSON format for that integration testing which is not likely what Users would want when converting Arrow to JSON.

There is not currently a recommended way to serialize an instance of arrow.Table to JSON because there is not standardized external format expected as JSON for arrow to my knowledge. Having arrjson be internal is intentional due to it being for the internal integration testing.

-----Original Message-----
From: Micah Kornfield <em...@gmail.com> 
Sent: Wednesday, April 21, 2021 11:19 PM
To: dev <de...@arrow.apache.org>
Subject: Re: [Go] expose ability to write arrow.Table to JSON

> I was wondering why `arrjson` is kept as an internal package within
go/arrow.

I think this is probably what is used for internal integration testing, we have specific JSON format that is expected, that needs to free to evolve and probably isn't what users are looking for.


> Is there a different recommended way to serialize an instance of
`arrow.Table` to JSON? afaict, using `arrjson.Writer` seems like a great way to do this, but it isn't possible to depend on it as an external user, because of it residing within the `.../internal/...` path.

Sorry I don't know the answer to this, but I would guess there isn't anything currently in the Go package for this.


On Wed, Apr 21, 2021 at 6:45 PM Agam Brahma <ag...@gmail.com> wrote:

> Hi all,
>
> I was wondering why `arrjson` is kept as an internal package within 
> go/arrow.
>
> Is there a different recommended way to serialize an instance of 
> `arrow.Table` to JSON? afaict, using `arrjson.Writer` seems like a 
> great way to do this, but it isn't possible to depend on it as an 
> external user, because of it residing within the `.../internal/...` path.
>
> Any pointers are much appreciated.
>
> Thanks,
> Agam
>
>
>
>

Re: [Go] expose ability to write arrow.Table to JSON

[Micah Kornfield <em...@gmail.com>]

> I was wondering why `arrjson` is kept as an internal package within
go/arrow.

I think this is probably what is used for internal integration testing, we
have specific JSON format that is expected, that needs to free to evolve
and probably isn't what users are looking for.


> Is there a different recommended way to serialize an instance of
`arrow.Table` to JSON? afaict, using `arrjson.Writer` seems like a great
way to do this, but it isn't possible to depend on it as an external user,
because of it residing within the `.../internal/...` path.

Sorry I don't know the answer to this, but I would guess there isn't
anything currently in the Go package for this.


On Wed, Apr 21, 2021 at 6:45 PM Agam Brahma <ag...@gmail.com> wrote:

> Hi all,
>
> I was wondering why `arrjson` is kept as an internal package within
> go/arrow.
>
> Is there a different recommended way to serialize an instance of
> `arrow.Table` to JSON? afaict, using `arrjson.Writer` seems like a great
> way to do this, but it isn't possible to depend on it as an external user,
> because of it residing within the `.../internal/...` path.
>
> Any pointers are much appreciated.
>
> Thanks,
> Agam
>
>
>
>