Format for Arrow Flight descriptor and schema for DoPut from Go

[Simon Knight <si...@gmail.com>]

Hi,
I have a simple Flight server in Go. I am able to receive data to a Go server from a Python client:

import pyarrow.flight as flight
import polars as pl
df = pl.read_csv("test.csv")
data_table = df.to_arrow()
client = pa.flight.connect("grpc://0.0.0.0:58474")
upload_descriptor = pa.flight.FlightDescriptor.for_path("uploaded.parquet")
writer, _ = client.do_put(upload_descriptor, data_table.schema)
writer.write_table(data_table)
writer.close()

This works perfectly.

But I am getting very stuck with the methods to format the DoPut to include the descriptor and schema for the Go client.

My approximate code is:

client, err := flight.NewClientWithMiddleware(address, nil,
nil, grpc.WithTransportCredentials(insecure.NewCredentials()))
stream, err := client.DoPut(context.Background())
if err != nil {
panic(err)
}

wr := flight.NewRecordWriter(stream)

descriptor := &flight.FlightDescriptor{
Type: flight.DescriptorPATH,
Path: []string{*output},
}

wr.SetFlightDescriptor(descriptor)
schema := flight.SerializeSchema(rr.Schema(), memory.DefaultAllocator)

defer rr.Release()

for rr.Next() {
rec := rr.Record()
print("rec", rec.NumRows())
err = wr.Write(rec)
}

My question is how to correctly insert the flight.FlightDescriptor and flight.SerializeSchema pieces.

The comments for these methods suggest it can be a bit intricate.

I can't find an example from the docs, the Flight SQL example in the repository, or docs such as https://voltrondata.com/resources/data-transfer-with-apache-arrow-and-golang or https://blog.djnavarro.net/posts/2022-10-18_arrow-flight/

All the examples I could find (blogs and tests) only show DoGet, where it appears the NewRecordWriter does most of the heavy lifting

Thanks

Thanks

Re: Format for Arrow Flight descriptor and schema for DoPut from Go

[Simon Knight <si...@gmail.com>]

Hi Bryce,
Thanks for the quick reply, that worked perfectly.

Thanks
On 14 Oct 2023 at 3:58 AM +1030, Bryce Mecum <br...@gmail.com>, wrote:
> Hi Simon, take a look at the highlighted portion in scenario.go [1]
> and see if that helps. I think it's pretty similar to what you're
> wanting to do.
>
> PS: Your email shows up as a mostly-white wall of text. To read it I
> had to paste it into a plain text editor.
>
> [1] https://github.com/apache/arrow/blob/e9aac8a4ce4b8833667efe52957fb0881a488c4b/go/arrow/internal/flight_integration/scenario.go#L193-L221
>
>
> On Fri, Oct 13, 2023 at 4:39 AM Simon Knight <si...@gmail.com> wrote:
> >
> > Hi,
> > I have a simple Flight server in Go. I am able to receive data to a Go server from a Python client:
> >
> > import pyarrow.flight as flight
> > import polars as pl
> > df = pl.read_csv("test.csv")
> > data_table = df.to_arrow()
> > client = pa.flight.connect("grpc://0.0.0.0:58474")
> > upload_descriptor = pa.flight.FlightDescriptor.for_path("uploaded.parquet")
> > writer, _ = client.do_put(upload_descriptor, data_table.schema)
> > writer.write_table(data_table)
> > writer.close()
> >
> > This works perfectly.
> >
> > But I am getting very stuck with the methods to format the DoPut to include the descriptor and schema for the Go client.
> >
> > My approximate code is:
> >
> > client, err := flight.NewClientWithMiddleware(address, nil,
> > nil, grpc.WithTransportCredentials(insecure.NewCredentials()))
> > stream, err := client.DoPut(context.Background())
> > if err != nil {
> > panic(err)
> > }
> >
> > wr := flight.NewRecordWriter(stream)
> >
> > descriptor := &flight.FlightDescriptor{
> > Type: flight.DescriptorPATH,
> > Path: []string{*output},
> > }
> >
> > wr.SetFlightDescriptor(descriptor)
> > schema := flight.SerializeSchema(rr.Schema(), memory.DefaultAllocator)
> >
> > defer rr.Release()
> >
> > for rr.Next() {
> > rec := rr.Record()
> > print("rec", rec.NumRows())
> > err = wr.Write(rec)
> > }
> >
> > My question is how to correctly insert the flight.FlightDescriptor and flight.SerializeSchema pieces.
> >
> > The comments for these methods suggest it can be a bit intricate.
> >
> > I can't find an example from the docs, the Flight SQL example in the repository, or docs such as https://voltrondata.com/resources/data-transfer-with-apache-arrow-and-golang or https://blog.djnavarro.net/posts/2022-10-18_arrow-flight/
> >
> > All the examples I could find (blogs and tests) only show DoGet, where it appears the NewRecordWriter does most of the heavy lifting
> >
> > Thanks
> >
> > Thanks

Re: Format for Arrow Flight descriptor and schema for DoPut from Go

[Bryce Mecum <br...@gmail.com>]

Hi Simon, take a look at the highlighted portion in scenario.go [1]
and see if that helps. I think it's pretty similar to what you're
wanting to do.

PS: Your email shows up as a mostly-white wall of text. To read it I
had to paste it into a plain text editor.

[1] https://github.com/apache/arrow/blob/e9aac8a4ce4b8833667efe52957fb0881a488c4b/go/arrow/internal/flight_integration/scenario.go#L193-L221


On Fri, Oct 13, 2023 at 4:39 AM Simon Knight <si...@gmail.com> wrote:
>
> Hi,
> I have a simple Flight server in Go. I am able to receive data to a Go server from a Python client:
>
> import pyarrow.flight as flight
> import polars as pl
> df = pl.read_csv("test.csv")
> data_table = df.to_arrow()
> client = pa.flight.connect("grpc://0.0.0.0:58474")
> upload_descriptor = pa.flight.FlightDescriptor.for_path("uploaded.parquet")
> writer, _ = client.do_put(upload_descriptor, data_table.schema)
> writer.write_table(data_table)
> writer.close()
>
> This works perfectly.
>
> But I am getting very stuck with the methods to format the DoPut to include the descriptor and schema for the Go client.
>
> My approximate code is:
>
> client, err := flight.NewClientWithMiddleware(address, nil,
> nil, grpc.WithTransportCredentials(insecure.NewCredentials()))
> stream, err := client.DoPut(context.Background())
> if err != nil {
> panic(err)
> }
>
> wr := flight.NewRecordWriter(stream)
>
> descriptor := &flight.FlightDescriptor{
> Type: flight.DescriptorPATH,
> Path: []string{*output},
> }
>
> wr.SetFlightDescriptor(descriptor)
> schema := flight.SerializeSchema(rr.Schema(), memory.DefaultAllocator)
>
> defer rr.Release()
>
> for rr.Next() {
> rec := rr.Record()
> print("rec", rec.NumRows())
> err = wr.Write(rec)
> }
>
> My question is how to correctly insert the flight.FlightDescriptor and flight.SerializeSchema pieces.
>
> The comments for these methods suggest it can be a bit intricate.
>
> I can't find an example from the docs, the Flight SQL example in the repository, or docs such as https://voltrondata.com/resources/data-transfer-with-apache-arrow-and-golang or https://blog.djnavarro.net/posts/2022-10-18_arrow-flight/
>
> All the examples I could find (blogs and tests) only show DoGet, where it appears the NewRecordWriter does most of the heavy lifting
>
> Thanks
>
> Thanks