You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by sozin <so...@gmail.com> on 2020/08/08 15:43:02 UTC
[Python] Providing user-defined meta-data to FlightClient.put()
Hello! I'm using the Python client library, and would like to include some
additional meta-data to the server during the do_put call. For example,
`lineage = dict(id=id,
created_by=user_name,
created_on=creation_date,
derived_from=[derived_from_id],
git_sha=git_sha)
client.do_put(flight_descriptor, schema, lineage)
`
(I have a requirement to track the data lineage for columnar-oriented data,
hence the meta-data contains things like who made the df, when they made
it, the gitsha of the program that made the data, etc).
I have gone through the flight descriptor code to see if it had anything to
help me (hoping for something generic like
`fight_descriptor.set_property("meta_data", meta_data)` that I could use
during the server's `do_put`), but found nothing :-(
Any thoughts on how I might accomplish this? Thanks in advance!
Re: [Python] Providing user-defined meta-data to FlightClient.put()
Posted by sozin <so...@gmail.com>.
Ah, didn't know that the schema class had a metadata member, thank you!
Just tried it out using the demo client and server provided in the
python/examples, and it worked like a charm. (Looks like `add_metadata` is
deprecated, and `with_metadata
On Sat, Aug 8, 2020 at 12:09 PM Wes McKinney <we...@gmail.com> wrote:
> You can put that metadata in the data schema, is there a reason that
> won't work?
>
> On Sat, Aug 8, 2020 at 10:43 AM sozin <so...@gmail.com> wrote:
> >
> > Hello! I'm using the Python client library, and would like to include
> some additional meta-data to the server during the do_put call. For example,
> >
> > `lineage = dict(id=id,
> > created_by=user_name,
> > created_on=creation_date,
> > derived_from=[derived_from_id],
> > git_sha=git_sha)
> > client.do_put(flight_descriptor, schema, lineage)
> > `
> > (I have a requirement to track the data lineage for columnar-oriented
> data, hence the meta-data contains things like who made the df, when they
> made it, the gitsha of the program that made the data, etc).
> >
> > I have gone through the flight descriptor code to see if it had anything
> to help me (hoping for something generic like
> `fight_descriptor.set_property("meta_data", meta_data)` that I could use
> during the server's `do_put`), but found nothing :-(
> >
> > Any thoughts on how I might accomplish this? Thanks in advance!
>
Re: [Python] Providing user-defined meta-data to FlightClient.put()
Posted by Wes McKinney <we...@gmail.com>.
You can put that metadata in the data schema, is there a reason that
won't work?
On Sat, Aug 8, 2020 at 10:43 AM sozin <so...@gmail.com> wrote:
>
> Hello! I'm using the Python client library, and would like to include some additional meta-data to the server during the do_put call. For example,
>
> `lineage = dict(id=id,
> created_by=user_name,
> created_on=creation_date,
> derived_from=[derived_from_id],
> git_sha=git_sha)
> client.do_put(flight_descriptor, schema, lineage)
> `
> (I have a requirement to track the data lineage for columnar-oriented data, hence the meta-data contains things like who made the df, when they made it, the gitsha of the program that made the data, etc).
>
> I have gone through the flight descriptor code to see if it had anything to help me (hoping for something generic like `fight_descriptor.set_property("meta_data", meta_data)` that I could use during the server's `do_put`), but found nothing :-(
>
> Any thoughts on how I might accomplish this? Thanks in advance!