You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "aiguofer (via GitHub)" <gi...@apache.org> on 2023/09/08 18:46:19 UTC
[GitHub] [arrow] aiguofer opened a new issue, #37635: [FlightRPC] FlightSQL: Expose custom query metadata
aiguofer opened a new issue, #37635:
URL: https://github.com/apache/arrow/issues/37635
### Describe the enhancement requested
We'd like to be able to expose custom metadata related to a given query through our Arrow Flight SQL service. For example, it'd be great to expose `queryId`, `queryCost`, or `queryType`, which could be useful to clients/users.
It seems like the existing ability to add [custom metadata](https://arrow.apache.org/docs/format/Columnar.html#custom-application-metadata) could be leveraged to accomplish this, but the exact mechanics seem a little unclear. The easiest thing would be to expose it via the `Schema` metadata since every `FlightInfo` already includes one, making it easy to expose through existing implementations. However, this seems a little hacky conceptually since the metadata could be associated with the query and not the results themselves (although metadata for the results also makes sense).
Another option could be to add a way to expose custom metadata directly through `FlightInfo`, but this would likely require major changes in every language implementation.
As far as how this gets exposed on the client side, we'd likely need changes to the JDBC and ADBC drivers.
For JDBC, there could be a few methods:
- `PreparedStatement.getCustomMetadata()`
- `ResultSet.getCustomMetadata()`
- `ResultSetMetaData.getCustomMetadata()`
Thoughts?
### Component(s)
FlightRPC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #37635: [FlightRPC] FlightSQL: Expose custom query metadata
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #37635:
URL: https://github.com/apache/arrow/issues/37635#issuecomment-1737279806
A contribution would be welcome. Or, @vibhatha @davisusanibar are one of you able to take this on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #37635: [FlightRPC] FlightSQL: Expose custom query metadata
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #37635:
URL: https://github.com/apache/arrow/issues/37635#issuecomment-1712117941
I can't speak for the Java implementation, but many FlightRPC implementations expose the `app_metadata` field on the `FlightData` messages during a stream which tends to be the common method for including this type of metadata (the other method tends to be including it in the schema of the result set, i.e. the "hacky" way you mentioned). Personally I prefer it being part of the `FlightData` as it ensures that every message is able to contain metadata which may potentially include things like `partitionId` and so on above and beyond the query level metadata.
@kou @lidavidm what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] aiguofer commented on issue #37635: [FlightRPC] FlightSQL: Expose custom query metadata
Posted by "aiguofer (via GitHub)" <gi...@apache.org>.
aiguofer commented on issue #37635:
URL: https://github.com/apache/arrow/issues/37635#issuecomment-1736553307
Awesome, thanks for the action on this!! I missed the vote since I wasn't signed up to the mailing list.
What would need to be done to expose this in Java?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FlightRPC] FlightSQL: Expose custom query metadata [arrow]
Posted by "aiguofer (via GitHub)" <gi...@apache.org>.
aiguofer commented on issue #37635:
URL: https://github.com/apache/arrow/issues/37635#issuecomment-1745530012
I looked around, but I'm not sure how generate the grpc stubs for java. Are there any docs on that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FlightRPC] FlightSQL: Expose custom query metadata [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #37635:
URL: https://github.com/apache/arrow/issues/37635#issuecomment-1747428927
Yes - please file issues for both of those.
For Flight SQL, perhaps specifying that it's always a message containing `map<string, google.protobuf.Value>` is fine (to let clients deal with it generically) (this is basically equivalent to specifying it as a JSON map)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #37635: [FlightRPC] FlightSQL: Expose custom query metadata
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #37635:
URL: https://github.com/apache/arrow/issues/37635#issuecomment-1712151276
That's a fair point, I'd be in favor of it. I can put together a PR and propose it on the mailing list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #37635: [FlightRPC] FlightSQL: Expose custom query metadata
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #37635:
URL: https://github.com/apache/arrow/issues/37635#issuecomment-1712182175
Sounds good, thanks.
I think ADBC can expose this easily now that we have `getOptionBytes` (and it will also be part of the metadata `executePartitions` anyways).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #37635: [FlightRPC] FlightSQL: Expose custom query metadata
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #37635:
URL: https://github.com/apache/arrow/issues/37635#issuecomment-1714598714
I'll put a PR together tomorrow for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #37635: [FlightRPC] FlightSQL: Expose custom query metadata
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #37635:
URL: https://github.com/apache/arrow/issues/37635#issuecomment-1712146541
FlightData isn't relevant here? We're talking about FlightInfo.
I've pushed for app_metadata on FlightInfo before (or possibly FlightEndpoint, or both) and I think we should just do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FlightRPC] FlightSQL: Expose custom query metadata [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade closed issue #37635: [FlightRPC] FlightSQL: Expose custom query metadata
URL: https://github.com/apache/arrow/issues/37635
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FlightRPC] FlightSQL: Expose custom query metadata [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #37635:
URL: https://github.com/apache/arrow/issues/37635#issuecomment-1745539127
they're generated during the build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FlightRPC] FlightSQL: Expose custom query metadata [arrow]
Posted by "aiguofer (via GitHub)" <gi...@apache.org>.
aiguofer commented on issue #37635:
URL: https://github.com/apache/arrow/issues/37635#issuecomment-1747423509
Ahh great, that makes sense.
So I see the `app_metatada` is a simple `ArrowBuf`. I'm sure for regular `Arrow Flight` we want to let users implement it however they want. For `Arrow Flight SQL` do we want to specify some type of Map like format? I'm not sure what's the best way to add this to `FlightInfo`.
Should I make a separate issue for exposing this in java? We'll need the ability to set it in the FlightServer and read it in the FlightClient. Additionally, for my usecase, we'll need to expose it through the JDBC driver.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #37635: [FlightRPC] FlightSQL: Expose custom query metadata
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #37635:
URL: https://github.com/apache/arrow/issues/37635#issuecomment-1712284107
I'm OK with adding `app_metadata` to `FlightInfo` (and `FlightEndpoint`) too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #37635: [FlightRPC] FlightSQL: Expose custom query metadata
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #37635:
URL: https://github.com/apache/arrow/issues/37635#issuecomment-1716746675
Vote thread: https://lists.apache.org/thread/113xm107ptn1t76txockp11oc7wlh0ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org