You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/03/05 15:53:52 UTC
[GitHub] [arrow] lidavidm opened a new pull request #12573: ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects
lidavidm opened a new pull request #12573:
URL: https://github.com/apache/arrow/pull/12573
Adds a Flight SQL documentation page that we can link to from the website.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #12573: ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#issuecomment-1059786910
I would like to add sequence diagrams, how do we feel about something like [sphinxcontrib-mermaid](https://pypi.org/project/sphinxcontrib-mermaid/)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #12573: ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#discussion_r822930319
##########
File path: docs/source/format/FlightSql.rst
##########
@@ -0,0 +1,163 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _flight-sql:
+
+================
+Arrow Flight SQL
+================
+
+Arrow Flight SQL is a protocol for interacting with SQL databases
+using the Arrow in-memory format and the :doc:`Flight RPC
+<./Flight>` framework.
+
+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints. Flight SQL
+clients wrap the underlying Flight client to provide methods for the
+new RPC methods described here.
+
+.. warning:: Flight SQL is **experimental** and changes to the
+ protocol may still be made.
+
+RPC Methods
+===========
+
+Flight SQL reuses the predefined RPC methods in Arrow Flight, and
+provides various commands that pair those methods with request/response
+messages defined via Protobuf (see below).
+
+SQL Metadata
+------------
+
+Flight SQL provides a variety of commands to fetch catalog metadata
+about the database server.
+
+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.
Review comment:
Hmm, we would probably need to have Sphinx index the Protobuf file somehow and then use Sphinx's cross-reference support. Unfortunately there doesn't seem to be such integration from a quick glance (and if we wanted to do it, we would need to create a custom Protobuf plugin or modify the existing one (which supports neither Doxygen nor Sphinx): https://github.com/pseudomuto/protoc-gen-doc/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12573: ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#issuecomment-1063249689
Benchmark runs are scheduled for baseline = 875af62cf2ae5827609a9a4cbf4e9b688b3008a2 and contender = 76b1403aeed52989fc3c6f82a808142a9b536532. 76b1403aeed52989fc3c6f82a808142a9b536532 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a6cce84b128c423d9f0815e559842643...b66dd26d3c42475685463bed096610aa/)
[Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/c0663e7bb9dc42cf9b0a8c44e2752c28...b362ee33d6a54903b659b39ce786c6e1/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/dced88bc04254463b4deec1a2d232052...40d99c8dc9094f1aa863dc7eb34f8625/)
[Finished :arrow_down:0.09% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/3d6cc6efdce54a6f8d1dc9a4a2117254...93c8bb4ef9574a7d917f311151b209ed/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12573: ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#issuecomment-1063249689
Benchmark runs are scheduled for baseline = 875af62cf2ae5827609a9a4cbf4e9b688b3008a2 and contender = 76b1403aeed52989fc3c6f82a808142a9b536532. 76b1403aeed52989fc3c6f82a808142a9b536532 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a6cce84b128c423d9f0815e559842643...b66dd26d3c42475685463bed096610aa/)
[Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/c0663e7bb9dc42cf9b0a8c44e2752c28...b362ee33d6a54903b659b39ce786c6e1/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/dced88bc04254463b4deec1a2d232052...40d99c8dc9094f1aa863dc7eb34f8625/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/3d6cc6efdce54a6f8d1dc9a4a2117254...93c8bb4ef9574a7d917f311151b209ed/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #12573: ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#discussion_r821681046
##########
File path: docs/source/format/FlightSql.rst
##########
@@ -0,0 +1,162 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _flight-sql:
+
+================
+Arrow Flight SQL
+================
+
+Arrow Flight SQL is a protocol for interacting with SQL databases
+using the Arrow in-memory format and the :doc:`Flight RPC
+<./Flight>` framework.
+
+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints. Flight SQL
+clients wrap the underlying Flight client to provide methods for the
+new RPC methods described here.
+
+.. warning:: Flight SQL is **experimental** and changes to the
+ protocol may still be made.
+
+RPC Methods
+===========
+
+Flight SQL reuses the predefined RPC methods in Arrow Flight, and
+provides various commands that pair those methods with request/response
+messages defined via Protobuf (see below).
+
+SQL Metadata
+------------
+
+Flight SQL provides a variety of commands to fetch catalog metadata
+about the database server.
+
+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.
+
+If the command is used with GetFlightInfo, the server will return a
+FlightInfo response. The client should then use the Ticket(s) in the
+FlightInfo with the DoGet RPC method to fetch a Arrow data containing
+the results of the command. In other words, metadata is returned as
+Arrow data, just like query results themselves.
+
+The schema returned by GetSchema or DoGet for a particular command is
+fixed according to the specification.
+
+``CommandGetCatalogs``
+ List the catalogs available in the database. The definition varies
+ by vendor.
+
+``CommandGetCrossReference``
+ List the foreign key columns in a given table that reference
+ columns in a given parent table.
+
+``CommandGetDbSchemas``
+ List the schemas (note: a grouping of tables, *not* an Arrow
+ schema) available in the database. The definition varies by
+ vendor.
+
+``CommandGetExportedKeys``
+ List foreign key columns that reference the primary key columns of
+ a given table.
+
+``CommandGetImportedKeys``
+ List foreign keys of a given table.
+
+``CommandGetPrimaryKeys``
+ List the primary keys of a given table.
+
+``CommandGetSqlInfo``
+ Fetch metadata about the database server and its supported SQL
+ features.
+
+``CommandGetTables``
+ List tables in the database.
+
+``CommandGetTableTypes``
+ List table types in the database. The list of types varies by
+ vendor.
+
+Query Execution
+---------------
+
+Flight SQL also provides commands to execute SQL queries and manage
+prepared statements.
+
+Many of these commands are also used with GetFlightInfo/GetSchema and
+work identically to the metadata methods above. Some of these commands
+can be used with the DoPut RPC method, but the command should still be
+encoded in the request FlightDescriptor in the same way.
+
+Commands beginning with "Action" are instead used with the DoAction
+RPC method, in which case the command should be packed into a
+google.protobuf.Any message, then serialized and packed into the
+``body`` of a Flight Action. Also, the ``type`` should be set to the
+command name (i.e. for ``ActionClosePreparedStatementRequest``, the
+``type`` should be ``ClosePreparedStatement``).
+
+``ActionClosePreparedStatementRequest``
+ Close a previously created prepared statement.
+
+``ActionCreatePreparedStatementRequest``
+ Create a new prepared statement for a SQL query.
+
+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.
+
+ When used with DoPut: binds parameter values to the prepared statement.
Review comment:
Ah, it was due to output parameters potentially changing the schema mid-stream. https://github.com/apache/arrow/pull/9368#discussion_r572941765
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #12573: ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#issuecomment-1063249689
Benchmark runs are scheduled for baseline = 875af62cf2ae5827609a9a4cbf4e9b688b3008a2 and contender = 76b1403aeed52989fc3c6f82a808142a9b536532. 76b1403aeed52989fc3c6f82a808142a9b536532 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a6cce84b128c423d9f0815e559842643...b66dd26d3c42475685463bed096610aa/)
[Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/c0663e7bb9dc42cf9b0a8c44e2752c28...b362ee33d6a54903b659b39ce786c6e1/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/dced88bc04254463b4deec1a2d232052...40d99c8dc9094f1aa863dc7eb34f8625/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/3d6cc6efdce54a6f8d1dc9a4a2117254...93c8bb4ef9574a7d917f311151b209ed/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #12573: ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#issuecomment-1059787241
CC @jduo if you'd like to take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #12573: ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#discussion_r822927469
##########
File path: docs/source/format/FlightSql.rst
##########
@@ -0,0 +1,163 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _flight-sql:
+
+================
+Arrow Flight SQL
+================
+
+Arrow Flight SQL is a protocol for interacting with SQL databases
+using the Arrow in-memory format and the :doc:`Flight RPC
+<./Flight>` framework.
+
+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints. Flight SQL
+clients wrap the underlying Flight client to provide methods for the
+new RPC methods described here.
+
+.. warning:: Flight SQL is **experimental** and changes to the
+ protocol may still be made.
+
+RPC Methods
+===========
+
+Flight SQL reuses the predefined RPC methods in Arrow Flight, and
+provides various commands that pair those methods with request/response
+messages defined via Protobuf (see below).
+
+SQL Metadata
+------------
+
+Flight SQL provides a variety of commands to fetch catalog metadata
+about the database server.
+
+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.
Review comment:
For the record, would there be an easy way to cross-reference the table definitions in the Flight protobuf file (such as `FlightDescriptor` here)? I guess not, but just asking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12573: ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#issuecomment-1063249689
Benchmark runs are scheduled for baseline = 875af62cf2ae5827609a9a4cbf4e9b688b3008a2 and contender = 76b1403aeed52989fc3c6f82a808142a9b536532. 76b1403aeed52989fc3c6f82a808142a9b536532 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a6cce84b128c423d9f0815e559842643...b66dd26d3c42475685463bed096610aa/)
[Finished :arrow_down:0.08% :arrow_up:0.04%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/c0663e7bb9dc42cf9b0a8c44e2752c28...b362ee33d6a54903b659b39ce786c6e1/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/dced88bc04254463b4deec1a2d232052...40d99c8dc9094f1aa863dc7eb34f8625/)
[Finished :arrow_down:0.09% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/3d6cc6efdce54a6f8d1dc9a4a2117254...93c8bb4ef9574a7d917f311151b209ed/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #12573: ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#discussion_r820235714
##########
File path: docs/source/format/FlightSql.rst
##########
@@ -0,0 +1,159 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _flight-sql:
+
+================
+Arrow Flight SQL
+================
+
+Arrow Flight SQL is a protocol for interacting with SQL databases
+using the Arrow in-memory format and the :doc:`Flight RPC
+<./Flight>` framework.
+
+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints.
Review comment:
Let's mention that Flight SQL clients decorate existing Flight client classes to add methods corresponding to the new RPC calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #12573: ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#discussion_r821681583
##########
File path: docs/source/format/FlightSql.rst
##########
@@ -0,0 +1,162 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _flight-sql:
+
+================
+Arrow Flight SQL
+================
+
+Arrow Flight SQL is a protocol for interacting with SQL databases
+using the Arrow in-memory format and the :doc:`Flight RPC
+<./Flight>` framework.
+
+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints. Flight SQL
+clients wrap the underlying Flight client to provide methods for the
+new RPC methods described here.
+
+.. warning:: Flight SQL is **experimental** and changes to the
+ protocol may still be made.
+
+RPC Methods
+===========
+
+Flight SQL reuses the predefined RPC methods in Arrow Flight, and
+provides various commands that pair those methods with request/response
+messages defined via Protobuf (see below).
+
+SQL Metadata
+------------
+
+Flight SQL provides a variety of commands to fetch catalog metadata
+about the database server.
+
+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.
+
+If the command is used with GetFlightInfo, the server will return a
+FlightInfo response. The client should then use the Ticket(s) in the
+FlightInfo with the DoGet RPC method to fetch a Arrow data containing
+the results of the command. In other words, metadata is returned as
+Arrow data, just like query results themselves.
+
+The schema returned by GetSchema or DoGet for a particular command is
+fixed according to the specification.
+
+``CommandGetCatalogs``
+ List the catalogs available in the database. The definition varies
+ by vendor.
+
+``CommandGetCrossReference``
+ List the foreign key columns in a given table that reference
+ columns in a given parent table.
+
+``CommandGetDbSchemas``
+ List the schemas (note: a grouping of tables, *not* an Arrow
+ schema) available in the database. The definition varies by
+ vendor.
+
+``CommandGetExportedKeys``
+ List foreign key columns that reference the primary key columns of
+ a given table.
+
+``CommandGetImportedKeys``
+ List foreign keys of a given table.
+
+``CommandGetPrimaryKeys``
+ List the primary keys of a given table.
+
+``CommandGetSqlInfo``
+ Fetch metadata about the database server and its supported SQL
+ features.
+
+``CommandGetTables``
+ List tables in the database.
+
+``CommandGetTableTypes``
+ List table types in the database. The list of types varies by
+ vendor.
+
+Query Execution
+---------------
+
+Flight SQL also provides commands to execute SQL queries and manage
+prepared statements.
+
+Many of these commands are also used with GetFlightInfo/GetSchema and
+work identically to the metadata methods above. Some of these commands
+can be used with the DoPut RPC method, but the command should still be
+encoded in the request FlightDescriptor in the same way.
+
+Commands beginning with "Action" are instead used with the DoAction
+RPC method, in which case the command should be packed into a
+google.protobuf.Any message, then serialized and packed into the
+``body`` of a Flight Action. Also, the ``type`` should be set to the
+command name (i.e. for ``ActionClosePreparedStatementRequest``, the
+``type`` should be ``ClosePreparedStatement``).
+
+``ActionClosePreparedStatementRequest``
+ Close a previously created prepared statement.
+
+``ActionCreatePreparedStatementRequest``
+ Create a new prepared statement for a SQL query.
+
+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.
Review comment:
It is reusable (else, you don't save much from having a prepared statement).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #12573: ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#issuecomment-1062979656
@jduo @pitrou any other comments here? I think we can use ARROW-14698 to expand the documentation once this lands
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm closed pull request #12573: ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
lidavidm closed pull request #12573:
URL: https://github.com/apache/arrow/pull/12573
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #12573: ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#issuecomment-1060906134
> I would like to add sequence diagrams, how do we feel about something like [sphinxcontrib-mermaid](https://pypi.org/project/sphinxcontrib-mermaid/)?
Looks interesting. @jorisvandenbossche What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #12573: ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#discussion_r821679736
##########
File path: docs/source/format/FlightSql.rst
##########
@@ -0,0 +1,162 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _flight-sql:
+
+================
+Arrow Flight SQL
+================
+
+Arrow Flight SQL is a protocol for interacting with SQL databases
+using the Arrow in-memory format and the :doc:`Flight RPC
+<./Flight>` framework.
+
+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints. Flight SQL
+clients wrap the underlying Flight client to provide methods for the
+new RPC methods described here.
+
+.. warning:: Flight SQL is **experimental** and changes to the
+ protocol may still be made.
+
+RPC Methods
+===========
+
+Flight SQL reuses the predefined RPC methods in Arrow Flight, and
+provides various commands that pair those methods with request/response
+messages defined via Protobuf (see below).
+
+SQL Metadata
+------------
+
+Flight SQL provides a variety of commands to fetch catalog metadata
+about the database server.
+
+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.
+
+If the command is used with GetFlightInfo, the server will return a
+FlightInfo response. The client should then use the Ticket(s) in the
+FlightInfo with the DoGet RPC method to fetch a Arrow data containing
+the results of the command. In other words, metadata is returned as
+Arrow data, just like query results themselves.
+
+The schema returned by GetSchema or DoGet for a particular command is
+fixed according to the specification.
+
+``CommandGetCatalogs``
+ List the catalogs available in the database. The definition varies
+ by vendor.
+
+``CommandGetCrossReference``
+ List the foreign key columns in a given table that reference
+ columns in a given parent table.
+
+``CommandGetDbSchemas``
+ List the schemas (note: a grouping of tables, *not* an Arrow
+ schema) available in the database. The definition varies by
+ vendor.
+
+``CommandGetExportedKeys``
+ List foreign key columns that reference the primary key columns of
+ a given table.
+
+``CommandGetImportedKeys``
+ List foreign keys of a given table.
+
+``CommandGetPrimaryKeys``
+ List the primary keys of a given table.
+
+``CommandGetSqlInfo``
+ Fetch metadata about the database server and its supported SQL
+ features.
+
+``CommandGetTables``
+ List tables in the database.
+
+``CommandGetTableTypes``
+ List table types in the database. The list of types varies by
+ vendor.
+
+Query Execution
+---------------
+
+Flight SQL also provides commands to execute SQL queries and manage
+prepared statements.
+
+Many of these commands are also used with GetFlightInfo/GetSchema and
+work identically to the metadata methods above. Some of these commands
+can be used with the DoPut RPC method, but the command should still be
+encoded in the request FlightDescriptor in the same way.
+
+Commands beginning with "Action" are instead used with the DoAction
+RPC method, in which case the command should be packed into a
+google.protobuf.Any message, then serialized and packed into the
+``body`` of a Flight Action. Also, the ``type`` should be set to the
+command name (i.e. for ``ActionClosePreparedStatementRequest``, the
+``type`` should be ``ClosePreparedStatement``).
+
+``ActionClosePreparedStatementRequest``
+ Close a previously created prepared statement.
+
+``ActionCreatePreparedStatementRequest``
+ Create a new prepared statement for a SQL query.
+
+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.
+
+ When used with DoPut: binds parameter values to the prepared statement.
Review comment:
I believe this was considered during the initial discussion but was rejected; @jduo do you remember why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #12573: ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#discussion_r820232922
##########
File path: docs/source/format/FlightSql.rst
##########
@@ -0,0 +1,159 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _flight-sql:
+
+================
+Arrow Flight SQL
+================
+
+Arrow Flight SQL is a protocol for interacting with SQL databases
+using the Arrow in-memory format and the :doc:`Flight RPC
+<./Flight>` framework.
+
+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints.
+
+.. warning:: Flight SQL is **experimental** and changes to the
+ protocol may still be made.
+
+RPC Methods
+===========
+
+Flight SQL reuses the predefined RPC methods in Arrow Flight, and
+provides various commands that pair those methods with request/response
+messages defined via Protobuf (see below).
+
+SQL Metadata
+------------
+
+Flight SQL provides a variety of commands to fetch metadata about the
+database server.
+
+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.
+
+If the command is used with GetFlightInfo, the server will return a
+FlightInfo response. The client should then use the Ticket(s) in the
+FlightInfo with the DoGet RPC method to fetch a Arrow data containing
+the results of the command. In other words, metadata is returned as
+Arrow data, just like query results themselves.
+
+The schema returned by GetSchema or DoGet for a particular command is
+fixed according to the specification.
+
+``CommandGetCatalogs``
+ List the catalogs available in the database. The definition varies
+ by vendor.
+
+``CommandGetCrossReference``
+ List the foreign key columns in a given table that reference
+ columns in a given parent table.
+
+``CommandGetDbSchemas``
+ List the schemas (note: *not* an Arrow schema) available in the
Review comment:
```suggestion
List the schemas (note: a grouping of tables, *not* an Arrow schema) available in the
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #12573: ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#issuecomment-1059786886
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #12573: ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#discussion_r820907182
##########
File path: docs/source/format/FlightSql.rst
##########
@@ -0,0 +1,162 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _flight-sql:
+
+================
+Arrow Flight SQL
+================
+
+Arrow Flight SQL is a protocol for interacting with SQL databases
+using the Arrow in-memory format and the :doc:`Flight RPC
+<./Flight>` framework.
+
+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints. Flight SQL
+clients wrap the underlying Flight client to provide methods for the
+new RPC methods described here.
+
+.. warning:: Flight SQL is **experimental** and changes to the
+ protocol may still be made.
+
+RPC Methods
+===========
+
+Flight SQL reuses the predefined RPC methods in Arrow Flight, and
+provides various commands that pair those methods with request/response
+messages defined via Protobuf (see below).
+
+SQL Metadata
+------------
+
+Flight SQL provides a variety of commands to fetch catalog metadata
+about the database server.
+
+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.
+
+If the command is used with GetFlightInfo, the server will return a
+FlightInfo response. The client should then use the Ticket(s) in the
+FlightInfo with the DoGet RPC method to fetch a Arrow data containing
+the results of the command. In other words, metadata is returned as
+Arrow data, just like query results themselves.
+
+The schema returned by GetSchema or DoGet for a particular command is
Review comment:
```suggestion
The Arrow schema returned by GetSchema or DoGet for a particular command is
```
##########
File path: docs/source/format/FlightSql.rst
##########
@@ -0,0 +1,162 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _flight-sql:
+
+================
+Arrow Flight SQL
+================
+
+Arrow Flight SQL is a protocol for interacting with SQL databases
+using the Arrow in-memory format and the :doc:`Flight RPC
+<./Flight>` framework.
+
+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints. Flight SQL
+clients wrap the underlying Flight client to provide methods for the
+new RPC methods described here.
+
+.. warning:: Flight SQL is **experimental** and changes to the
+ protocol may still be made.
+
+RPC Methods
+===========
+
+Flight SQL reuses the predefined RPC methods in Arrow Flight, and
+provides various commands that pair those methods with request/response
+messages defined via Protobuf (see below).
+
+SQL Metadata
+------------
+
+Flight SQL provides a variety of commands to fetch catalog metadata
+about the database server.
+
+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.
+
+If the command is used with GetFlightInfo, the server will return a
+FlightInfo response. The client should then use the Ticket(s) in the
+FlightInfo with the DoGet RPC method to fetch a Arrow data containing
+the results of the command. In other words, metadata is returned as
Review comment:
```suggestion
the results of the command. In other words, SQL metadata is returned as
```
##########
File path: docs/source/format/FlightSql.rst
##########
@@ -0,0 +1,162 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _flight-sql:
+
+================
+Arrow Flight SQL
+================
+
+Arrow Flight SQL is a protocol for interacting with SQL databases
+using the Arrow in-memory format and the :doc:`Flight RPC
+<./Flight>` framework.
+
+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints. Flight SQL
+clients wrap the underlying Flight client to provide methods for the
+new RPC methods described here.
+
+.. warning:: Flight SQL is **experimental** and changes to the
+ protocol may still be made.
+
+RPC Methods
+===========
+
+Flight SQL reuses the predefined RPC methods in Arrow Flight, and
+provides various commands that pair those methods with request/response
+messages defined via Protobuf (see below).
+
+SQL Metadata
+------------
+
+Flight SQL provides a variety of commands to fetch catalog metadata
+about the database server.
+
+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.
+
+If the command is used with GetFlightInfo, the server will return a
+FlightInfo response. The client should then use the Ticket(s) in the
+FlightInfo with the DoGet RPC method to fetch a Arrow data containing
+the results of the command. In other words, metadata is returned as
+Arrow data, just like query results themselves.
+
+The schema returned by GetSchema or DoGet for a particular command is
+fixed according to the specification.
+
+``CommandGetCatalogs``
+ List the catalogs available in the database. The definition varies
+ by vendor.
+
+``CommandGetCrossReference``
+ List the foreign key columns in a given table that reference
+ columns in a given parent table.
+
+``CommandGetDbSchemas``
+ List the schemas (note: a grouping of tables, *not* an Arrow
+ schema) available in the database. The definition varies by
+ vendor.
+
+``CommandGetExportedKeys``
+ List foreign key columns that reference the primary key columns of
+ a given table.
+
+``CommandGetImportedKeys``
+ List foreign keys of a given table.
+
+``CommandGetPrimaryKeys``
+ List the primary keys of a given table.
+
+``CommandGetSqlInfo``
+ Fetch metadata about the database server and its supported SQL
+ features.
+
+``CommandGetTables``
+ List tables in the database.
+
+``CommandGetTableTypes``
+ List table types in the database. The list of types varies by
+ vendor.
+
+Query Execution
+---------------
+
+Flight SQL also provides commands to execute SQL queries and manage
+prepared statements.
+
+Many of these commands are also used with GetFlightInfo/GetSchema and
+work identically to the metadata methods above. Some of these commands
+can be used with the DoPut RPC method, but the command should still be
+encoded in the request FlightDescriptor in the same way.
+
+Commands beginning with "Action" are instead used with the DoAction
+RPC method, in which case the command should be packed into a
+google.protobuf.Any message, then serialized and packed into the
+``body`` of a Flight Action. Also, the ``type`` should be set to the
Review comment:
```suggestion
``body`` of a Flight Action. Also, the action ``type`` should be set to the
```
##########
File path: docs/source/format/FlightSql.rst
##########
@@ -0,0 +1,162 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _flight-sql:
+
+================
+Arrow Flight SQL
+================
+
+Arrow Flight SQL is a protocol for interacting with SQL databases
+using the Arrow in-memory format and the :doc:`Flight RPC
+<./Flight>` framework.
+
+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints. Flight SQL
+clients wrap the underlying Flight client to provide methods for the
+new RPC methods described here.
+
+.. warning:: Flight SQL is **experimental** and changes to the
+ protocol may still be made.
+
+RPC Methods
+===========
+
+Flight SQL reuses the predefined RPC methods in Arrow Flight, and
+provides various commands that pair those methods with request/response
+messages defined via Protobuf (see below).
+
+SQL Metadata
+------------
+
+Flight SQL provides a variety of commands to fetch catalog metadata
+about the database server.
+
+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.
+
+If the command is used with GetFlightInfo, the server will return a
+FlightInfo response. The client should then use the Ticket(s) in the
+FlightInfo with the DoGet RPC method to fetch a Arrow data containing
+the results of the command. In other words, metadata is returned as
+Arrow data, just like query results themselves.
+
+The schema returned by GetSchema or DoGet for a particular command is
+fixed according to the specification.
+
+``CommandGetCatalogs``
+ List the catalogs available in the database. The definition varies
+ by vendor.
+
+``CommandGetCrossReference``
+ List the foreign key columns in a given table that reference
+ columns in a given parent table.
+
+``CommandGetDbSchemas``
+ List the schemas (note: a grouping of tables, *not* an Arrow
+ schema) available in the database. The definition varies by
+ vendor.
+
+``CommandGetExportedKeys``
+ List foreign key columns that reference the primary key columns of
+ a given table.
+
+``CommandGetImportedKeys``
+ List foreign keys of a given table.
+
+``CommandGetPrimaryKeys``
+ List the primary keys of a given table.
+
+``CommandGetSqlInfo``
+ Fetch metadata about the database server and its supported SQL
+ features.
+
+``CommandGetTables``
+ List tables in the database.
+
+``CommandGetTableTypes``
+ List table types in the database. The list of types varies by
+ vendor.
+
+Query Execution
+---------------
+
+Flight SQL also provides commands to execute SQL queries and manage
+prepared statements.
+
+Many of these commands are also used with GetFlightInfo/GetSchema and
+work identically to the metadata methods above. Some of these commands
+can be used with the DoPut RPC method, but the command should still be
+encoded in the request FlightDescriptor in the same way.
+
+Commands beginning with "Action" are instead used with the DoAction
+RPC method, in which case the command should be packed into a
+google.protobuf.Any message, then serialized and packed into the
+``body`` of a Flight Action. Also, the ``type`` should be set to the
+command name (i.e. for ``ActionClosePreparedStatementRequest``, the
+``type`` should be ``ClosePreparedStatement``).
+
+``ActionClosePreparedStatementRequest``
+ Close a previously created prepared statement.
+
+``ActionCreatePreparedStatementRequest``
+ Create a new prepared statement for a SQL query.
+
+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.
+
+ When used with DoPut: binds parameter values to the prepared statement.
Review comment:
So one must call first `DoPut` then `GetFlightInfo` to execute a query with parameters? Wouldn't it be nicer to be able to use `DoExchange` for that?
##########
File path: docs/source/format/FlightSql.rst
##########
@@ -0,0 +1,162 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _flight-sql:
+
+================
+Arrow Flight SQL
+================
+
+Arrow Flight SQL is a protocol for interacting with SQL databases
+using the Arrow in-memory format and the :doc:`Flight RPC
+<./Flight>` framework.
+
+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints. Flight SQL
+clients wrap the underlying Flight client to provide methods for the
+new RPC methods described here.
+
+.. warning:: Flight SQL is **experimental** and changes to the
+ protocol may still be made.
+
+RPC Methods
+===========
+
+Flight SQL reuses the predefined RPC methods in Arrow Flight, and
+provides various commands that pair those methods with request/response
+messages defined via Protobuf (see below).
+
+SQL Metadata
+------------
+
+Flight SQL provides a variety of commands to fetch catalog metadata
+about the database server.
+
+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.
+
+If the command is used with GetFlightInfo, the server will return a
+FlightInfo response. The client should then use the Ticket(s) in the
+FlightInfo with the DoGet RPC method to fetch a Arrow data containing
+the results of the command. In other words, metadata is returned as
+Arrow data, just like query results themselves.
+
+The schema returned by GetSchema or DoGet for a particular command is
+fixed according to the specification.
+
+``CommandGetCatalogs``
+ List the catalogs available in the database. The definition varies
+ by vendor.
+
+``CommandGetCrossReference``
+ List the foreign key columns in a given table that reference
+ columns in a given parent table.
+
+``CommandGetDbSchemas``
+ List the schemas (note: a grouping of tables, *not* an Arrow
+ schema) available in the database. The definition varies by
+ vendor.
+
+``CommandGetExportedKeys``
+ List foreign key columns that reference the primary key columns of
+ a given table.
+
+``CommandGetImportedKeys``
+ List foreign keys of a given table.
+
+``CommandGetPrimaryKeys``
+ List the primary keys of a given table.
+
+``CommandGetSqlInfo``
+ Fetch metadata about the database server and its supported SQL
+ features.
+
+``CommandGetTables``
+ List tables in the database.
+
+``CommandGetTableTypes``
+ List table types in the database. The list of types varies by
+ vendor.
+
+Query Execution
+---------------
+
+Flight SQL also provides commands to execute SQL queries and manage
+prepared statements.
+
+Many of these commands are also used with GetFlightInfo/GetSchema and
+work identically to the metadata methods above. Some of these commands
+can be used with the DoPut RPC method, but the command should still be
+encoded in the request FlightDescriptor in the same way.
+
+Commands beginning with "Action" are instead used with the DoAction
+RPC method, in which case the command should be packed into a
+google.protobuf.Any message, then serialized and packed into the
+``body`` of a Flight Action. Also, the ``type`` should be set to the
+command name (i.e. for ``ActionClosePreparedStatementRequest``, the
+``type`` should be ``ClosePreparedStatement``).
+
+``ActionClosePreparedStatementRequest``
+ Close a previously created prepared statement.
+
+``ActionCreatePreparedStatementRequest``
+ Create a new prepared statement for a SQL query.
+
+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.
Review comment:
Does this also close the prepared statement or can it be reused multiple times?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #12573: ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#issuecomment-1060667361
Thanks for the suggestions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #12573: ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#discussion_r822930715
##########
File path: docs/source/format/FlightSql.rst
##########
@@ -0,0 +1,163 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _flight-sql:
+
+================
+Arrow Flight SQL
+================
+
+Arrow Flight SQL is a protocol for interacting with SQL databases
+using the Arrow in-memory format and the :doc:`Flight RPC
+<./Flight>` framework.
+
+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints. Flight SQL
+clients wrap the underlying Flight client to provide methods for the
+new RPC methods described here.
+
+.. warning:: Flight SQL is **experimental** and changes to the
+ protocol may still be made.
+
+RPC Methods
+===========
+
+Flight SQL reuses the predefined RPC methods in Arrow Flight, and
+provides various commands that pair those methods with request/response
+messages defined via Protobuf (see below).
+
+SQL Metadata
+------------
+
+Flight SQL provides a variety of commands to fetch catalog metadata
+about the database server.
+
+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.
Review comment:
That may be worthwhile, though, since then we could avoid repeating docs between both files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12573: ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#issuecomment-1063249689
Benchmark runs are scheduled for baseline = 875af62cf2ae5827609a9a4cbf4e9b688b3008a2 and contender = 76b1403aeed52989fc3c6f82a808142a9b536532. 76b1403aeed52989fc3c6f82a808142a9b536532 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a6cce84b128c423d9f0815e559842643...b66dd26d3c42475685463bed096610aa/)
[Finished :arrow_down:0.08% :arrow_up:0.04%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/c0663e7bb9dc42cf9b0a8c44e2752c28...b362ee33d6a54903b659b39ce786c6e1/)
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/dced88bc04254463b4deec1a2d232052...40d99c8dc9094f1aa863dc7eb34f8625/)
[Finished :arrow_down:0.09% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/3d6cc6efdce54a6f8d1dc9a4a2117254...93c8bb4ef9574a7d917f311151b209ed/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #12573: ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#discussion_r820232286
##########
File path: docs/source/format/FlightSql.rst
##########
@@ -0,0 +1,159 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _flight-sql:
+
+================
+Arrow Flight SQL
+================
+
+Arrow Flight SQL is a protocol for interacting with SQL databases
+using the Arrow in-memory format and the :doc:`Flight RPC
+<./Flight>` framework.
+
+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints.
+
+.. warning:: Flight SQL is **experimental** and changes to the
+ protocol may still be made.
+
+RPC Methods
+===========
+
+Flight SQL reuses the predefined RPC methods in Arrow Flight, and
+provides various commands that pair those methods with request/response
+messages defined via Protobuf (see below).
+
+SQL Metadata
+------------
+
+Flight SQL provides a variety of commands to fetch metadata about the
Review comment:
```suggestion
Flight SQL provides a variety of commands to fetch catalog metadata about the
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] removed a comment on pull request #12573: ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects
Posted by GitBox <gi...@apache.org>.
github-actions[bot] removed a comment on pull request #12573:
URL: https://github.com/apache/arrow/pull/12573#issuecomment-1059786888
:warning: Ticket **has not been started in JIRA**, please click 'Start Progress'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org