You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/01/30 02:13:36 UTC
[GitHub] [arrow] tifflhl opened a new pull request #9368: [WIP] [POC] Flight SQL
tifflhl opened a new pull request #9368:
URL: https://github.com/apache/arrow/pull/9368
**Introduction**
This proof-of-concept in progress illustrates how Flight SQL can be implemented. This follows designs proposed in the Flight SQL proposal socialized last year. You can find the original proposal [here](https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI/edit?usp=sharing). Since this is still a proof-of-concept, design feedback and suggestions are very much appreciated.
**An Overview of this PR**
- This PR adds a new module within Flight called flight-sql.
- flight-sql is a new Flight API that provides a standard way for clients and servers to communicate with SQL-like semantics.
- Like other Flight APIs, flight-sql does not provide _implementation details_ that dictate how a client and server communicates with each other, it simply provides the SQL semantics and apply them onto the Flight API.
**A Walkthrough of the New Module**
- FlightSQL.proto introduces new SQL protobuf objects.
- FlightSQLProducer introduces a new FlightProducer API that adapts classic Flight requests into SQL operations.
- FlighSQLExample is a sample FlightSQL server implementation.
As a proof-of-concept, this isn't complete yet. There are a few TODO items:
- FlightSQLExample is not completed yet.
- As of now, all flight-sql client related operations are I FlightSQLClientUtils. I would like to break this utility class apart and offer an API for a FlightSQLClient add-on.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-819722798
Following up here @tifflhl, is there interest in trying to get this in for 5.0.0 perhaps?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r635398800
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
Review comment:
IMO, RPC protocol version is what matters here from a 'capabilities-discovery' perspective, though the Arrow library release (including the commit hash) would be good as a separate field for debugging purposes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664142839
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
Review comment:
The typeIds aren't actually Flatbuffer type IDs - they're user-chosen sentinel values used to distinguish the cases on the wire; they're the tag of a tagged union. So they can be whatever. The actual types of the children are defined by the child fields, like so: https://github.com/apache/arrow/blob/41c4143992905cc85eb61a417cf9460c6db6b4df/java/vector/src/test/java/org/apache/arrow/vector/TestDenseUnionVector.java#L328-L343
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664914178
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSqlUtils.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.sql.Types;
+import java.util.List;
+
+import org.apache.arrow.flight.ActionType;
+import org.apache.arrow.vector.types.DateUnit;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+
+import com.google.common.collect.ImmutableList;
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+import com.google.protobuf.Message;
+
+/**
+ * Utilities to work with Flight SQL semantics.
+ */
+public final class FlightSqlUtils {
+
+ private static final int BIT_WIDTH8 = 8;
+ private static final int BIT_WIDTH_16 = 16;
+ private static final int BIT_WIDTH_32 = 32;
+ private static final int BIT_WIDTH_64 = 64;
+ private static final boolean IS_SIGNED_FALSE = false;
+ private static final boolean IS_SIGNED_TRUE = true;
+
+ public static final ActionType FLIGHT_SQL_CREATEPREPAREDSTATEMENT = new ActionType("CreatePreparedStatement",
+ "Creates a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionCreatePreparedStatementRequest\n" +
+ "Response Message: ActionCreatePreparedStatementResult");
+
+ public static final ActionType FLIGHT_SQL_CLOSEPREPAREDSTATEMENT = new ActionType("ClosePreparedStatement",
+ "Closes a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionClosePreparedStatementRequest\n" +
+ "Response Message: N/A");
+
+ public static final List<ActionType> FLIGHT_SQL_ACTIONS = ImmutableList.of(
+ FLIGHT_SQL_CREATEPREPAREDSTATEMENT,
+ FLIGHT_SQL_CLOSEPREPAREDSTATEMENT
+ );
+
+ /**
+ * Converts {@link java.sql.Types} values returned from JDBC Apis to Arrow types.
+ *
+ * @param jdbcDataType {@link java.sql.Types} value.
+ * @param precision Precision of the type.
+ * @param scale Scale of the type.
+ * @return The Arrow equivalent type.
+ */
+ public static ArrowType getArrowTypeFromJDBCType(int jdbcDataType, int precision, int scale) {
+
+ switch (jdbcDataType) {
+ case Types.BIT:
+ case Types.BOOLEAN:
+ return ArrowType.Bool.INSTANCE;
+ case Types.TINYINT:
+ return new ArrowType.Int(BIT_WIDTH8, IS_SIGNED_TRUE);
+ case Types.SMALLINT:
+ return new ArrowType.Int(BIT_WIDTH_16, IS_SIGNED_TRUE);
+ case Types.INTEGER:
+ return new ArrowType.Int(BIT_WIDTH_32, IS_SIGNED_TRUE);
+ case Types.BIGINT:
+ return new ArrowType.Int(BIT_WIDTH_64, IS_SIGNED_TRUE);
+ case Types.FLOAT:
+ case Types.REAL:
+ return new ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE);
+ case Types.DOUBLE:
+ return new ArrowType.FloatingPoint(FloatingPointPrecision.DOUBLE);
+ case Types.NUMERIC:
+ case Types.DECIMAL:
+ return new ArrowType.Decimal(precision, scale);
+ case Types.DATE:
+ return new ArrowType.Date(DateUnit.DAY);
+ case Types.TIME:
+ return new ArrowType.Time(TimeUnit.MILLISECOND, BIT_WIDTH_32);
+ case Types.TIMESTAMP:
+ return new ArrowType.Timestamp(TimeUnit.MILLISECOND, null);
+ case Types.BINARY:
+ case Types.VARBINARY:
+ case Types.LONGVARBINARY:
+ return ArrowType.Binary.INSTANCE;
+ case Types.NULL:
+ return ArrowType.Null.INSTANCE;
+
+ case Types.CHAR:
+ case Types.VARCHAR:
+ case Types.LONGVARCHAR:
+ case Types.CLOB:
+ case Types.NCHAR:
+ case Types.NVARCHAR:
+ case Types.LONGNVARCHAR:
+ case Types.NCLOB:
+
+ case Types.OTHER:
+ case Types.JAVA_OBJECT:
+ case Types.DISTINCT:
+ case Types.STRUCT:
+ case Types.ARRAY:
+ case Types.BLOB:
+ case Types.REF:
+ case Types.DATALINK:
+ case Types.ROWID:
+ case Types.SQLXML:
+ case Types.REF_CURSOR:
+ case Types.TIME_WITH_TIMEZONE:
+ case Types.TIMESTAMP_WITH_TIMEZONE:
+ default:
+ return ArrowType.Utf8.INSTANCE;
+ }
+ }
+
+ /**
+ * Helper to parse {@link com.google.protobuf.Any} objects to the specific protobuf object.
+ *
+ * @param source the raw bytes source value.
+ * @return the materialized protobuf object.
+ */
+ public static Any parseOrThrow(byte[] source) {
Review comment:
I think these helpers should take in optional ServerStreamListeners.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r567492292
##########
File path: java/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/PreparedStatementContext.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.util.Objects;
+
+import org.apache.arrow.util.AutoCloseables;
+
+class PreparedStatementContext implements AutoCloseable {
Review comment:
I think instead of placing this in tests, we should have a separate top level package for a JDBC implementation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-875051056
@lidavidm @wesm - how should we try to move this towards getting merged - do you want one PR (this one) where the Java client/server are there, and the test has a working example, or should we try and split it up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-770164877
Small style nit: Generally camel casing even for acronyms is preferred.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r637165066
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
Review comment:
We should expand the set of meta-tables to include more than just table discovery. For parity with ODBC/JDBC applications we should support tables that report primary key constraints, foreign key relations, statistics, and table privileges. I would consider this beyond of an MVP though.
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
Review comment:
We should expand the set of meta-tables to include more than just table discovery. For parity with ODBC/JDBC applications we should support tables that report primary key constraints, foreign key relations, statistics, and table privileges. I would consider this beyond an MVP though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r652306377
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
Review comment:
yes, if we can just use information_schema that seems reasonable to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] tifflhl commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
tifflhl commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-819915361
Hi David,
Our resources are a bit limited right now and can't really say if we would
have the bandwidth to work on flight-sql for v5.0.0. When we have some more
clearance on when the work can resume, we will definitely notify the dev
mailing list.
Cheers,
Tiffany
On Wed, Apr 14, 2021 at 11:03 AM David Li ***@***.***> wrote:
> Following up here @tifflhl <https://github.com/tifflhl>, is there
> interest in trying to get this in for 5.0.0 perhaps?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/arrow/pull/9368#issuecomment-819722798>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB35GBV4LK7XPD6SBK7VEWLTIXKGPANCNFSM4WZ2VVYA>
> .
>
--
Tiffany Lam (She/Her)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-881595452
@lidavidm - just as an update, work is progressing here on a separate branch that'll be merged in. I think we'll ultimately end up with a separate PR as Tiffany has left, so we can't alter this. Will put a link to the new PR once created here for tracing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r635398800
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
Review comment:
IMO, RPC protocol version is what matters here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r567204981
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
Review comment:
Also if these types aren't used initially maybe leave them out for now?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664136841
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
Review comment:
@lidavidm - how do you actually create a FieldType for a dense union like this? I have the following:
`new ArrowType.Union(UnionMode.Dense, new int[2] {ArrowTypeID.Utf8, ArrowTypeID.Int}), /*dictionary=*/null)`, although I could not find an ArrowTypeID for a bigint, and it seems repetitive to have the int_value and int32_bitmask.
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
Review comment:
Slightly revised, but I think still incorrect: `new ArrowType.Union(UnionMode.Dense, new int[] {Type.Utf8, Type.Int})`
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
Review comment:
Thanks - I think I have it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-917713477
Closing in favor of #10906
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664927502
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSqlUtils.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.sql.Types;
+import java.util.List;
+
+import org.apache.arrow.flight.ActionType;
+import org.apache.arrow.vector.types.DateUnit;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+
+import com.google.common.collect.ImmutableList;
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+import com.google.protobuf.Message;
+
+/**
+ * Utilities to work with Flight SQL semantics.
+ */
+public final class FlightSqlUtils {
+
+ private static final int BIT_WIDTH8 = 8;
+ private static final int BIT_WIDTH_16 = 16;
+ private static final int BIT_WIDTH_32 = 32;
+ private static final int BIT_WIDTH_64 = 64;
+ private static final boolean IS_SIGNED_FALSE = false;
+ private static final boolean IS_SIGNED_TRUE = true;
+
+ public static final ActionType FLIGHT_SQL_CREATEPREPAREDSTATEMENT = new ActionType("CreatePreparedStatement",
+ "Creates a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionCreatePreparedStatementRequest\n" +
+ "Response Message: ActionCreatePreparedStatementResult");
+
+ public static final ActionType FLIGHT_SQL_CLOSEPREPAREDSTATEMENT = new ActionType("ClosePreparedStatement",
+ "Closes a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionClosePreparedStatementRequest\n" +
+ "Response Message: N/A");
+
+ public static final List<ActionType> FLIGHT_SQL_ACTIONS = ImmutableList.of(
+ FLIGHT_SQL_CREATEPREPAREDSTATEMENT,
+ FLIGHT_SQL_CLOSEPREPAREDSTATEMENT
+ );
+
+ /**
+ * Converts {@link java.sql.Types} values returned from JDBC Apis to Arrow types.
+ *
+ * @param jdbcDataType {@link java.sql.Types} value.
+ * @param precision Precision of the type.
+ * @param scale Scale of the type.
+ * @return The Arrow equivalent type.
+ */
+ public static ArrowType getArrowTypeFromJDBCType(int jdbcDataType, int precision, int scale) {
Review comment:
This looks like it's against old code - this is now within the example code only.
But, I see the `JdbcToArrowConfig` which looks like it does what we want if we can leverage it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield closed pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield closed pull request #9368:
URL: https://github.com/apache/arrow/pull/9368
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664911275
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSqlProducer.java
##########
@@ -0,0 +1,541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.arrow.flight.Action;
+import org.apache.arrow.flight.ActionType;
+import org.apache.arrow.flight.FlightDescriptor;
+import org.apache.arrow.flight.FlightInfo;
+import org.apache.arrow.flight.FlightProducer;
+import org.apache.arrow.flight.FlightStream;
+import org.apache.arrow.flight.PutResult;
+import org.apache.arrow.flight.Result;
+import org.apache.arrow.flight.SchemaResult;
+import org.apache.arrow.flight.Ticket;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionClosePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionCreatePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetCatalogs;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetSchemas;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetSqlInfo;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetTableTypes;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetTables;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementQuery;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementUpdate;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandStatementQuery;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandStatementUpdate;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.UnionMode;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.types.pojo.Schema;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+
+import io.grpc.Status;
+
+/**
+ * API to Implement an Arrow Flight SQL producer.
+ */
+public abstract class FlightSqlProducer implements FlightProducer, AutoCloseable {
+ /**
+ * Depending on the provided command, method either:
+ * 1. Return information about a SQL query, or
+ * 2. Return information about a prepared statement. In this case, parameters binding is allowed.
+ *
+ * @param context Per-call context.
+ * @param descriptor The descriptor identifying the data stream.
+ * @return information about the given SQL query, or the given prepared statement.
+ */
+ @Override
+ public FlightInfo getFlightInfo(CallContext context, FlightDescriptor descriptor) {
+ final Any command = FlightSqlUtils.parseOrThrow(descriptor.getCommand());
+
+ if (command.is(CommandStatementQuery.class)) {
+ return getFlightInfoStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandStatementQuery.class), context, descriptor);
+ } else if (command.is(CommandPreparedStatementQuery.class)) {
+ return getFlightInfoPreparedStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandPreparedStatementQuery.class), context, descriptor);
+ } else if (command.is(CommandGetCatalogs.class)) {
+ return getFlightInfoCatalogs(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetCatalogs.class), context, descriptor);
+ } else if (command.is(CommandGetSchemas.class)) {
+ return getFlightInfoSchemas(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetSchemas.class), context, descriptor);
+ } else if (command.is(CommandGetTables.class)) {
+ return getFlightInfoTables(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetTables.class), context, descriptor);
+ } else if (command.is(CommandGetTableTypes.class)) {
+ return getFlightInfoTableTypes(context, descriptor);
+ } else if (command.is(CommandGetSqlInfo.class)) {
+ return getFlightInfoSqlInfo(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetSqlInfo.class), context, descriptor);
+ }
+
+ throw Status.INVALID_ARGUMENT.asRuntimeException();
+ }
+
+ /**
+ * Returns the schema of the result produced by the SQL query.
+ *
+ * @param context Per-call context.
+ * @param descriptor The descriptor identifying the data stream.
+ * @return the result set schema.
+ */
+ @Override
+ public SchemaResult getSchema(CallContext context, FlightDescriptor descriptor) {
+ final Any command = FlightSqlUtils.parseOrThrow(descriptor.getCommand());
+
+ if (command.is(CommandStatementQuery.class)) {
+ return getSchemaStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandStatementQuery.class), context, descriptor);
+ } else if (command.is(CommandGetCatalogs.class)) {
+ return getSchemaCatalogs();
+ } else if (command.is(CommandGetSchemas.class)) {
+ return getSchemaSchemas();
+ } else if (command.is(CommandGetTables.class)) {
+ return getSchemaTables();
+ } else if (command.is(CommandGetTableTypes.class)) {
+ return getSchemaTableTypes();
+ } else if (command.is(CommandGetSqlInfo.class)) {
+ return getSchemaSqlInfo();
+ }
+
+ throw Status.INVALID_ARGUMENT.asRuntimeException();
+ }
+
+ /**
+ * Depending on the provided command, method either:
+ * 1. Return data for a stream produced by executing the provided SQL query, or
+ * 2. Return data for a prepared statement. In this case, parameters binding is allowed.
+ *
+ * @param context Per-call context.
+ * @param ticket The application-defined ticket identifying this stream.
+ * @param listener An interface for sending data back to the client.
+ */
+ @Override
+ public void getStream(CallContext context, Ticket ticket, ServerStreamListener listener) {
+ final Any command;
+
+ try {
+ command = Any.parseFrom(ticket.getBytes());
+ } catch (InvalidProtocolBufferException e) {
+ listener.error(e);
+ return;
+ }
+
+ if (command.is(CommandStatementQuery.class)) {
+ getStreamStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandStatementQuery.class), context, ticket, listener);
+ } else if (command.is(CommandPreparedStatementQuery.class)) {
+ getStreamPreparedStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandPreparedStatementQuery.class), context, ticket, listener);
+ } else if (command.is(CommandGetCatalogs.class)) {
+ getStreamCatalogs(context, ticket, listener);
+ } else if (command.is(CommandGetSchemas.class)) {
+ getStreamSchemas(FlightSqlUtils.unpackOrThrow(command, CommandGetSchemas.class), context, ticket, listener);
+ } else if (command.is(CommandGetTables.class)) {
+ getStreamTables(FlightSqlUtils.unpackOrThrow(command, CommandGetTables.class), context, ticket, listener);
+ } else if (command.is(CommandGetTableTypes.class)) {
+ getStreamTableTypes(context, ticket, listener);
+ } else if (command.is(CommandGetSqlInfo.class)) {
+ getStreamSqlInfo(FlightSqlUtils.unpackOrThrow(command, CommandGetSqlInfo.class), context, ticket, listener);
+ } else {
+ throw Status.INVALID_ARGUMENT.asRuntimeException();
Review comment:
This needs to get set on the listener (the server can log this error though).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-770138862
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW
Then could you also rename pull request title in the following format?
ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r634805540
##########
File path: java/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/PreparedStatementContext.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.util.Objects;
+
+import org.apache.arrow.util.AutoCloseables;
+
+class PreparedStatementContext implements AutoCloseable {
Review comment:
I agree with this. Let's create a separate module FlightSQLForJDBC or something similar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r567204923
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
Review comment:
Is this also modelled after ODBC? is long_bitmask not preferred?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664911800
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSqlProducer.java
##########
@@ -0,0 +1,541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.arrow.flight.Action;
+import org.apache.arrow.flight.ActionType;
+import org.apache.arrow.flight.FlightDescriptor;
+import org.apache.arrow.flight.FlightInfo;
+import org.apache.arrow.flight.FlightProducer;
+import org.apache.arrow.flight.FlightStream;
+import org.apache.arrow.flight.PutResult;
+import org.apache.arrow.flight.Result;
+import org.apache.arrow.flight.SchemaResult;
+import org.apache.arrow.flight.Ticket;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionClosePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionCreatePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetCatalogs;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetSchemas;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetSqlInfo;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetTableTypes;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetTables;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementQuery;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementUpdate;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandStatementQuery;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandStatementUpdate;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.UnionMode;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.types.pojo.Schema;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+
+import io.grpc.Status;
+
+/**
+ * API to Implement an Arrow Flight SQL producer.
+ */
+public abstract class FlightSqlProducer implements FlightProducer, AutoCloseable {
+ /**
+ * Depending on the provided command, method either:
+ * 1. Return information about a SQL query, or
+ * 2. Return information about a prepared statement. In this case, parameters binding is allowed.
+ *
+ * @param context Per-call context.
+ * @param descriptor The descriptor identifying the data stream.
+ * @return information about the given SQL query, or the given prepared statement.
+ */
+ @Override
+ public FlightInfo getFlightInfo(CallContext context, FlightDescriptor descriptor) {
+ final Any command = FlightSqlUtils.parseOrThrow(descriptor.getCommand());
+
+ if (command.is(CommandStatementQuery.class)) {
+ return getFlightInfoStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandStatementQuery.class), context, descriptor);
+ } else if (command.is(CommandPreparedStatementQuery.class)) {
+ return getFlightInfoPreparedStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandPreparedStatementQuery.class), context, descriptor);
+ } else if (command.is(CommandGetCatalogs.class)) {
+ return getFlightInfoCatalogs(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetCatalogs.class), context, descriptor);
+ } else if (command.is(CommandGetSchemas.class)) {
+ return getFlightInfoSchemas(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetSchemas.class), context, descriptor);
+ } else if (command.is(CommandGetTables.class)) {
+ return getFlightInfoTables(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetTables.class), context, descriptor);
+ } else if (command.is(CommandGetTableTypes.class)) {
+ return getFlightInfoTableTypes(context, descriptor);
+ } else if (command.is(CommandGetSqlInfo.class)) {
+ return getFlightInfoSqlInfo(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetSqlInfo.class), context, descriptor);
+ }
+
+ throw Status.INVALID_ARGUMENT.asRuntimeException();
+ }
+
+ /**
+ * Returns the schema of the result produced by the SQL query.
+ *
+ * @param context Per-call context.
+ * @param descriptor The descriptor identifying the data stream.
+ * @return the result set schema.
+ */
+ @Override
+ public SchemaResult getSchema(CallContext context, FlightDescriptor descriptor) {
+ final Any command = FlightSqlUtils.parseOrThrow(descriptor.getCommand());
+
+ if (command.is(CommandStatementQuery.class)) {
+ return getSchemaStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandStatementQuery.class), context, descriptor);
+ } else if (command.is(CommandGetCatalogs.class)) {
+ return getSchemaCatalogs();
+ } else if (command.is(CommandGetSchemas.class)) {
+ return getSchemaSchemas();
+ } else if (command.is(CommandGetTables.class)) {
+ return getSchemaTables();
+ } else if (command.is(CommandGetTableTypes.class)) {
+ return getSchemaTableTypes();
+ } else if (command.is(CommandGetSqlInfo.class)) {
+ return getSchemaSqlInfo();
+ }
+
+ throw Status.INVALID_ARGUMENT.asRuntimeException();
+ }
+
+ /**
+ * Depending on the provided command, method either:
+ * 1. Return data for a stream produced by executing the provided SQL query, or
+ * 2. Return data for a prepared statement. In this case, parameters binding is allowed.
+ *
+ * @param context Per-call context.
+ * @param ticket The application-defined ticket identifying this stream.
+ * @param listener An interface for sending data back to the client.
+ */
+ @Override
+ public void getStream(CallContext context, Ticket ticket, ServerStreamListener listener) {
+ final Any command;
+
+ try {
+ command = Any.parseFrom(ticket.getBytes());
+ } catch (InvalidProtocolBufferException e) {
+ listener.error(e);
+ return;
+ }
+
+ if (command.is(CommandStatementQuery.class)) {
+ getStreamStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandStatementQuery.class), context, ticket, listener);
+ } else if (command.is(CommandPreparedStatementQuery.class)) {
+ getStreamPreparedStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandPreparedStatementQuery.class), context, ticket, listener);
+ } else if (command.is(CommandGetCatalogs.class)) {
+ getStreamCatalogs(context, ticket, listener);
+ } else if (command.is(CommandGetSchemas.class)) {
+ getStreamSchemas(FlightSqlUtils.unpackOrThrow(command, CommandGetSchemas.class), context, ticket, listener);
+ } else if (command.is(CommandGetTables.class)) {
+ getStreamTables(FlightSqlUtils.unpackOrThrow(command, CommandGetTables.class), context, ticket, listener);
+ } else if (command.is(CommandGetTableTypes.class)) {
+ getStreamTableTypes(context, ticket, listener);
+ } else if (command.is(CommandGetSqlInfo.class)) {
+ getStreamSqlInfo(FlightSqlUtils.unpackOrThrow(command, CommandGetSqlInfo.class), context, ticket, listener);
+ } else {
+ throw Status.INVALID_ARGUMENT.asRuntimeException();
Review comment:
All calls to unpackOrThrow need to be in a try/catch that writes the result to the listener, the client will hang.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r660892357
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
+
+ // Implementation specific custom SqlInfo values.
+ repeated string custom_info = 2;
+}
+
+/*
+ Represents a request to retrieve the list of catalogs on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name.
+ */
+message CommandGetCatalogs {
+}
+
+/*
+ * Represents a request to retrieve the list of schemas on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8
+ * >
Review comment:
Just a thought, but it might be better to have `schema_name: list<utf8>` since otherwise, you'll be duplicating the `catalog_name` value for every `schema_name` value. (Or else we should dictionary-encode catalog names.)
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
+
+ // Implementation specific custom SqlInfo values.
+ repeated string custom_info = 2;
+}
+
+/*
+ Represents a request to retrieve the list of catalogs on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name.
+ */
+message CommandGetCatalogs {
+}
+
+/*
+ * Represents a request to retrieve the list of schemas on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name, then schema_name.
+ */
+message CommandGetSchemas {
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then all catalogs are searched.
+ */
+ string catalog = 1;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 2;
+}
+
+/*
+ * Represents a request to retrieve the list of tables, and optionally their schemas, on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8,
+ * table_name: utf8,
+ * table_type: utf8,
+ * table_schema: bytes
Review comment:
nit: let's specify exactly what the schema is (IPC-encapsulated serialized Flatbuffer schema). And ditto my comment above about duplicated string data - though I suppose, it might not be a big deal so long as the number of schemas/tables is relatively small?
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
Review comment:
It's not specified above, but is this now a list of metadata fields to retrieve?
Are there enough metadata fields that this is even worth filtering?
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
Review comment:
Should this be `CommandGetTables` now?
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
+
+ // Implementation specific custom SqlInfo values.
+ repeated string custom_info = 2;
+}
+
+/*
+ Represents a request to retrieve the list of catalogs on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name.
+ */
+message CommandGetCatalogs {
+}
+
+/*
+ * Represents a request to retrieve the list of schemas on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name, then schema_name.
+ */
+message CommandGetSchemas {
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then all catalogs are searched.
+ */
+ string catalog = 1;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 2;
+}
+
+/*
+ * Represents a request to retrieve the list of tables, and optionally their schemas, on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8,
+ * table_name: utf8,
+ * table_type: utf8,
+ * table_schema: bytes
+ * >
+ * The returned data should be ordered by catalog_name, schema_name, table_name, then table_type.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then all catalogs are searched.
+ */
+ string catalog = 1;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, all schemas matching other filters are searched.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 2;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no table_name_filter_pattern is provided, all tables matching other filters are searched.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 3;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 4;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 5;
+}
+
+/*
+ * Represents a request to retrieve the list of table types on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * table_type: utf8
+ * >
+ * The returned data should be ordered by table_type.
+ */
+message CommandGetTableTypes {
+}
Review comment:
For all commands, is the request flow now
1. Call `GetFlightInfo(FlightDescriptor(cmd=(serialized command)))`
2. Call `DoGet(flight_info.endpoints[0].ticket)`
?
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
+
+ // Implementation specific custom SqlInfo values.
+ repeated string custom_info = 2;
+}
+
+/*
+ Represents a request to retrieve the list of catalogs on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name.
+ */
+message CommandGetCatalogs {
+}
+
+/*
+ * Represents a request to retrieve the list of schemas on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name, then schema_name.
+ */
+message CommandGetSchemas {
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then all catalogs are searched.
+ */
+ string catalog = 1;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 2;
+}
+
+/*
+ * Represents a request to retrieve the list of tables, and optionally their schemas, on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8,
+ * table_name: utf8,
+ * table_type: utf8,
+ * table_schema: bytes
+ * >
+ * The returned data should be ordered by catalog_name, schema_name, table_name, then table_type.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then all catalogs are searched.
+ */
+ string catalog = 1;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, all schemas matching other filters are searched.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 2;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no table_name_filter_pattern is provided, all tables matching other filters are searched.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 3;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 4;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 5;
+}
+
+/*
+ * Represents a request to retrieve the list of table types on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * table_type: utf8
+ * >
+ * The returned data should be ordered by table_type.
+ */
+message CommandGetTableTypes {
+}
+
+// SQL Execution Action Messages
+
+/*
+ * Request message for the "GetPreparedStatement" action on a Flight SQL enabled backend.
+ */
+message ActionCreatePreparedStatementRequest {
+ // The valid SQL string to create a prepared statement for.
+ string query = 1;
+}
+
+/*
+ * Wrap the result of a "GetPreparedStatement" action.
+ */
+message ActionCreatePreparedStatementResult {
+ // Opaque handle for the prepared statement on the server.
+ bytes prepared_statement_handle = 1;
+
+ // If a result set generating query was provided, dataset_schema contains the
+ // schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ bytes dataset_schema = 2;
+
+ // If the query provided contained parameters, parameter_schema contains the
+ // Schema of the expected parameters as described in Schema.fbs::Schema.
Review comment:
nit: both of these should say something like `IPC-encapsulated serialized Flatbuffer schema` for consistency
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
+
+ // Implementation specific custom SqlInfo values.
+ repeated string custom_info = 2;
+}
+
+/*
+ Represents a request to retrieve the list of catalogs on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name.
+ */
+message CommandGetCatalogs {
+}
+
+/*
+ * Represents a request to retrieve the list of schemas on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name, then schema_name.
+ */
+message CommandGetSchemas {
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then all catalogs are searched.
+ */
+ string catalog = 1;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 2;
+}
+
+/*
+ * Represents a request to retrieve the list of tables, and optionally their schemas, on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8,
+ * table_name: utf8,
+ * table_type: utf8,
+ * table_schema: bytes
+ * >
+ * The returned data should be ordered by catalog_name, schema_name, table_name, then table_type.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then all catalogs are searched.
+ */
+ string catalog = 1;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, all schemas matching other filters are searched.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 2;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no table_name_filter_pattern is provided, all tables matching other filters are searched.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 3;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 4;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 5;
+}
+
+/*
+ * Represents a request to retrieve the list of table types on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * table_type: utf8
+ * >
+ * The returned data should be ordered by table_type.
+ */
+message CommandGetTableTypes {
+}
+
+// SQL Execution Action Messages
+
+/*
+ * Request message for the "GetPreparedStatement" action on a Flight SQL enabled backend.
+ */
+message ActionCreatePreparedStatementRequest {
+ // The valid SQL string to create a prepared statement for.
+ string query = 1;
+}
+
+/*
+ * Wrap the result of a "GetPreparedStatement" action.
+ */
+message ActionCreatePreparedStatementResult {
+ // Opaque handle for the prepared statement on the server.
+ bytes prepared_statement_handle = 1;
+
+ // If a result set generating query was provided, dataset_schema contains the
+ // schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ bytes dataset_schema = 2;
+
+ // If the query provided contained parameters, parameter_schema contains the
+ // Schema of the expected parameters as described in Schema.fbs::Schema.
+ bytes parameter_schema = 3;
+}
+
+/*
+ * Request message for the "ClosePreparedStatement" action on a Flight SQL enabled backend.
+ * Closes server resources associated with the prepared statement handle.
+ */
+message ActionClosePreparedStatementRequest {
+ // Opaque handle for the prepared statement on the server.
+ string prepared_statement_handle = 1;
+}
+
+
+// SQL Execution Messages.
+
+/*
+ * Represents a SQL query. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ */
+message CommandStatementQuery {
+ // The SQL syntax.
+ string query = 1;
+}
+
+/*
+ * Represents an instance of executing a prepared statement. Used in the command member of FlightDescriptor for
+ * the following RPC calls:
+ * - DoPut: bind parameter values.
+ * - GetFlightInfo: execute the prepared statement instance.
+ */
Review comment:
There used to be a TODO here about what the semantics of this were when binding multiple rows of parameter values with DoPut - was that resolved?
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
+
+ // Implementation specific custom SqlInfo values.
+ repeated string custom_info = 2;
+}
+
+/*
+ Represents a request to retrieve the list of catalogs on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name.
+ */
+message CommandGetCatalogs {
+}
+
+/*
+ * Represents a request to retrieve the list of schemas on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8
+ * >
Review comment:
(Though, see below; we duplicated the data before in Protobuf as well and likely it's not so much data that we need to worry about it.)
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
+
+ // Implementation specific custom SqlInfo values.
+ repeated string custom_info = 2;
+}
+
+/*
+ Represents a request to retrieve the list of catalogs on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name.
+ */
+message CommandGetCatalogs {
+}
+
+/*
+ * Represents a request to retrieve the list of schemas on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name, then schema_name.
+ */
+message CommandGetSchemas {
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then all catalogs are searched.
+ */
+ string catalog = 1;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 2;
+}
+
+/*
+ * Represents a request to retrieve the list of tables, and optionally their schemas, on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8,
+ * table_name: utf8,
+ * table_type: utf8,
+ * table_schema: bytes
Review comment:
And I suppose the data was also duplicated before in Protobuf. So probably not worth worrying as it would make the schema much less convenient to work with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664142839
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
Review comment:
The typeIds aren't actually Flatbuffer type IDs - they're user-chosen sentinel values used to distinguish the cases on the wire; they're the tag of a tagged union. So they can be whatever. The actual types of the children are defined by the child fields, like so: https://github.com/apache/arrow/blob/41c4143992905cc85eb61a417cf9460c6db6b4df/java/vector/src/test/java/org/apache/arrow/vector/TestDenseUnionVector.java#L328-L343
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
Review comment:
It took me a while to grok unions initially. The docs have a page explaining the layout: https://arrow.apache.org/docs/format/Columnar.html#dense-union
It might be clearer to think of type id as type discriminant in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-868657093
Thanks for picking this up again. Note the 5.0.0 timeline is soon-ish. It looks like we are nearing consensus on most issues raised so it would be helpful to see an updated document that could perhaps be submitted for a vote.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664928107
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSqlUtils.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.sql.Types;
+import java.util.List;
+
+import org.apache.arrow.flight.ActionType;
+import org.apache.arrow.vector.types.DateUnit;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+
+import com.google.common.collect.ImmutableList;
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+import com.google.protobuf.Message;
+
+/**
+ * Utilities to work with Flight SQL semantics.
+ */
+public final class FlightSqlUtils {
+
+ private static final int BIT_WIDTH8 = 8;
+ private static final int BIT_WIDTH_16 = 16;
+ private static final int BIT_WIDTH_32 = 32;
+ private static final int BIT_WIDTH_64 = 64;
+ private static final boolean IS_SIGNED_FALSE = false;
+ private static final boolean IS_SIGNED_TRUE = true;
+
+ public static final ActionType FLIGHT_SQL_CREATEPREPAREDSTATEMENT = new ActionType("CreatePreparedStatement",
+ "Creates a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionCreatePreparedStatementRequest\n" +
+ "Response Message: ActionCreatePreparedStatementResult");
+
+ public static final ActionType FLIGHT_SQL_CLOSEPREPAREDSTATEMENT = new ActionType("ClosePreparedStatement",
+ "Closes a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionClosePreparedStatementRequest\n" +
+ "Response Message: N/A");
+
+ public static final List<ActionType> FLIGHT_SQL_ACTIONS = ImmutableList.of(
+ FLIGHT_SQL_CREATEPREPAREDSTATEMENT,
+ FLIGHT_SQL_CLOSEPREPAREDSTATEMENT
+ );
+
+ /**
+ * Converts {@link java.sql.Types} values returned from JDBC Apis to Arrow types.
+ *
+ * @param jdbcDataType {@link java.sql.Types} value.
+ * @param precision Precision of the type.
+ * @param scale Scale of the type.
+ * @return The Arrow equivalent type.
+ */
+ public static ArrowType getArrowTypeFromJDBCType(int jdbcDataType, int precision, int scale) {
+
+ switch (jdbcDataType) {
+ case Types.BIT:
+ case Types.BOOLEAN:
+ return ArrowType.Bool.INSTANCE;
+ case Types.TINYINT:
+ return new ArrowType.Int(BIT_WIDTH8, IS_SIGNED_TRUE);
+ case Types.SMALLINT:
+ return new ArrowType.Int(BIT_WIDTH_16, IS_SIGNED_TRUE);
+ case Types.INTEGER:
+ return new ArrowType.Int(BIT_WIDTH_32, IS_SIGNED_TRUE);
+ case Types.BIGINT:
+ return new ArrowType.Int(BIT_WIDTH_64, IS_SIGNED_TRUE);
+ case Types.FLOAT:
+ case Types.REAL:
+ return new ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE);
+ case Types.DOUBLE:
+ return new ArrowType.FloatingPoint(FloatingPointPrecision.DOUBLE);
+ case Types.NUMERIC:
+ case Types.DECIMAL:
+ return new ArrowType.Decimal(precision, scale);
+ case Types.DATE:
+ return new ArrowType.Date(DateUnit.DAY);
+ case Types.TIME:
+ return new ArrowType.Time(TimeUnit.MILLISECOND, BIT_WIDTH_32);
+ case Types.TIMESTAMP:
+ return new ArrowType.Timestamp(TimeUnit.MILLISECOND, null);
+ case Types.BINARY:
+ case Types.VARBINARY:
+ case Types.LONGVARBINARY:
+ return ArrowType.Binary.INSTANCE;
+ case Types.NULL:
+ return ArrowType.Null.INSTANCE;
+
+ case Types.CHAR:
+ case Types.VARCHAR:
+ case Types.LONGVARCHAR:
+ case Types.CLOB:
+ case Types.NCHAR:
+ case Types.NVARCHAR:
+ case Types.LONGNVARCHAR:
+ case Types.NCLOB:
+
+ case Types.OTHER:
+ case Types.JAVA_OBJECT:
+ case Types.DISTINCT:
+ case Types.STRUCT:
+ case Types.ARRAY:
+ case Types.BLOB:
+ case Types.REF:
+ case Types.DATALINK:
+ case Types.ROWID:
+ case Types.SQLXML:
+ case Types.REF_CURSOR:
+ case Types.TIME_WITH_TIMEZONE:
Review comment:
Doesn't seem to exist?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r652296037
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
Review comment:
@emkornfield - rather than define new pseudo tables, would it be better to align to what's already in the INFORMATION_SCHEMA?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664927617
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSqlUtils.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.sql.Types;
+import java.util.List;
+
+import org.apache.arrow.flight.ActionType;
+import org.apache.arrow.vector.types.DateUnit;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+
+import com.google.common.collect.ImmutableList;
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+import com.google.protobuf.Message;
+
+/**
+ * Utilities to work with Flight SQL semantics.
+ */
+public final class FlightSqlUtils {
+
+ private static final int BIT_WIDTH8 = 8;
+ private static final int BIT_WIDTH_16 = 16;
+ private static final int BIT_WIDTH_32 = 32;
+ private static final int BIT_WIDTH_64 = 64;
+ private static final boolean IS_SIGNED_FALSE = false;
+ private static final boolean IS_SIGNED_TRUE = true;
+
+ public static final ActionType FLIGHT_SQL_CREATEPREPAREDSTATEMENT = new ActionType("CreatePreparedStatement",
+ "Creates a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionCreatePreparedStatementRequest\n" +
+ "Response Message: ActionCreatePreparedStatementResult");
+
+ public static final ActionType FLIGHT_SQL_CLOSEPREPAREDSTATEMENT = new ActionType("ClosePreparedStatement",
+ "Closes a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionClosePreparedStatementRequest\n" +
+ "Response Message: N/A");
+
+ public static final List<ActionType> FLIGHT_SQL_ACTIONS = ImmutableList.of(
+ FLIGHT_SQL_CREATEPREPAREDSTATEMENT,
+ FLIGHT_SQL_CLOSEPREPAREDSTATEMENT
+ );
+
+ /**
+ * Converts {@link java.sql.Types} values returned from JDBC Apis to Arrow types.
+ *
+ * @param jdbcDataType {@link java.sql.Types} value.
+ * @param precision Precision of the type.
+ * @param scale Scale of the type.
+ * @return The Arrow equivalent type.
+ */
+ public static ArrowType getArrowTypeFromJDBCType(int jdbcDataType, int precision, int scale) {
+
+ switch (jdbcDataType) {
+ case Types.BIT:
+ case Types.BOOLEAN:
+ return ArrowType.Bool.INSTANCE;
+ case Types.TINYINT:
+ return new ArrowType.Int(BIT_WIDTH8, IS_SIGNED_TRUE);
+ case Types.SMALLINT:
+ return new ArrowType.Int(BIT_WIDTH_16, IS_SIGNED_TRUE);
+ case Types.INTEGER:
+ return new ArrowType.Int(BIT_WIDTH_32, IS_SIGNED_TRUE);
+ case Types.BIGINT:
+ return new ArrowType.Int(BIT_WIDTH_64, IS_SIGNED_TRUE);
+ case Types.FLOAT:
+ case Types.REAL:
+ return new ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE);
+ case Types.DOUBLE:
+ return new ArrowType.FloatingPoint(FloatingPointPrecision.DOUBLE);
+ case Types.NUMERIC:
+ case Types.DECIMAL:
+ return new ArrowType.Decimal(precision, scale);
+ case Types.DATE:
+ return new ArrowType.Date(DateUnit.DAY);
+ case Types.TIME:
+ return new ArrowType.Time(TimeUnit.MILLISECOND, BIT_WIDTH_32);
Review comment:
Probably to be ignored if we're using the `arrow-jdbc` package.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664143005
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
Review comment:
It took me a while to grok unions initially. The docs have a page explaining the layout: https://arrow.apache.org/docs/format/Columnar.html#dense-union
It might be clearer to think of type id as type discriminant in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-881601040
@kylepbit Sounds good, thanks for the update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r658934729
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 6;
+}
+
+/*
+ * Wrap the result of a "GetTables" action.
+ */
+message ActionGetTablesResult {
+ string catalog = 1;
+ string schema = 2;
+ string table = 3;
+ string table_type = 4;
+
+ /*
+ * Schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ * Null if include_schema on request is false.
+ */
+ bytes arrow_metadata = 5;
+}
+
+/*
+ * Wrap the result of a "GetTableTypes" action.
+ */
+message ActionGetTableTypesResult {
+ /*
+ * Indicates the type of the table. E.g. table (regular data table) , view, system table etc.
+ */
+ string table_type = 1;
+}
+
+// SQL Execution Action Messages
+
+/*
+ * Request message for the "GetPreparedStatement" action on a
+ * Flight SQL enabled backend.
+ */
+message ActionGetPreparedStatementRequest {
+ // The valid SQL string to get a prepared statement for.
+ string query = 1;
+}
+
+/*
+ * Wrap the result of a "GetPreparedStatement" action.
+ */
+message ActionGetPreparedStatementResult {
+
+ // Opaque handle for the prepared statement on the server.
+ bytes prepared_statement_handle = 1;
+
+ // If a result set generating query was provided, dataset_schema contains the
+ // schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ bytes dataset_schema = 2;
+
+ // If the query provided contained parameters, parameter_schema contains the
+ // Schema of the expected parameters as described in Schema.fbs::Schema.
+ bytes parameter_schema = 3;
+}
+
+/*
+ * Request message for the "ClosePreparedStatement" action on a
+ * Flight SQL enabled backend.
+ * Closes server resources associated with the prepared statement handle.
+ */
+message ActionClosePreparedStatementRequest {
+ // Opaque handle for the prepared statement on the server.
+ string prepared_statement_handle = 1;
+}
+
+
+// SQL Execution Messages.
+
+/*
+ * Represents a SQL query. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ */
+message CommandStatementQuery {
+ // The SQL syntax.
+ string query = 1;
+}
+
+/*
+ * Represents an instance of executing a prepared statement. Used in the
+ * command member of FlightDescriptor for the following RPC calls:
+ * TODO: Is the idea that a Put with parameter values would execute multiple bound versions of the prepared statement
Review comment:
Ah sorry if I wasn't clear - yes, that sounds good, we can add output parameters hopefully soon afterward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-776035415
Thinking about this a bit more I think an alternative to a lot of the metadata querying is to define a minimal sql table schema and querying capabilities for it. I'll expand more in a little bit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r660924710
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
Review comment:
Yeah - I'll update the intent here. If omitted then retrieve all.
There aren't enough at this point, but there will be as the list of metadata can be quite long. Additionally, some may become expensive to calculate at some point for some data sources, so it would be nice to be able to limit what you were asking for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r660925948
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
+
+ // Implementation specific custom SqlInfo values.
+ repeated string custom_info = 2;
+}
+
+/*
+ Represents a request to retrieve the list of catalogs on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name.
+ */
+message CommandGetCatalogs {
+}
+
+/*
+ * Represents a request to retrieve the list of schemas on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name, then schema_name.
+ */
+message CommandGetSchemas {
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then all catalogs are searched.
+ */
+ string catalog = 1;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 2;
+}
+
+/*
+ * Represents a request to retrieve the list of tables, and optionally their schemas, on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8,
+ * table_name: utf8,
+ * table_type: utf8,
+ * table_schema: bytes
Review comment:
Ack - will leave the duplication as is for now.
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
Review comment:
Yes - oversight.
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
+
+ // Implementation specific custom SqlInfo values.
+ repeated string custom_info = 2;
+}
+
+/*
+ Represents a request to retrieve the list of catalogs on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name.
+ */
+message CommandGetCatalogs {
+}
+
+/*
+ * Represents a request to retrieve the list of schemas on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name, then schema_name.
+ */
+message CommandGetSchemas {
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then all catalogs are searched.
+ */
+ string catalog = 1;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 2;
+}
+
+/*
+ * Represents a request to retrieve the list of tables, and optionally their schemas, on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8,
+ * table_name: utf8,
+ * table_type: utf8,
+ * table_schema: bytes
+ * >
+ * The returned data should be ordered by catalog_name, schema_name, table_name, then table_type.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then all catalogs are searched.
+ */
+ string catalog = 1;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, all schemas matching other filters are searched.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 2;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no table_name_filter_pattern is provided, all tables matching other filters are searched.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 3;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 4;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 5;
+}
+
+/*
+ * Represents a request to retrieve the list of table types on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * table_type: utf8
+ * >
+ * The returned data should be ordered by table_type.
+ */
+message CommandGetTableTypes {
+}
Review comment:
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r567490946
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
Review comment:
```suggestion
* Specifies the order of result values with precendence:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-875052704
I think it's OK to have one PR with the format changes and a working client/server and test (that's how Flight itself was originally added, looking at it: https://github.com/apache/arrow/commit/db0ef22dd68ae00e11f09da40b6734c1d9770b57). Then we can put it up for a vote. Having the implementation will be useful so people voting can evaluate it as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664914977
##########
File path: java/flight/flight-sql/src/test/java/org/apache/arrow/flight/TestFlightSql.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight;
+
+import static org.apache.arrow.flight.sql.FlightSqlClientUtils.getPreparedStatement;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.arrow.flight.sql.FlightSqlClientUtils;
+import org.apache.arrow.flight.sql.FlightSqlExample;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionClosePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionCreatePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionCreatePreparedStatementResult;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementQuery;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.memory.util.ArrowBufPointer;
+import org.apache.arrow.util.AutoCloseables;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.arrow.vector.util.ElementAddressableVectorIterator;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.ByteString;
+
+/**
+ * Test direct usage of Flight SQL workflows.
+ */
+public class TestFlightSql {
+ private static BufferAllocator allocator;
+ private static FlightServer server;
+
+ private static FlightClient client;
+
+ protected static final Schema SCHEMA_INT_TABLE = new Schema(Arrays.asList(
+ new Field("KEYNAME", new
+ FieldType(true, ArrowType.Utf8.INSTANCE, null),
+ null),
+ new Field("VALUE",
+ new FieldType(true, new ArrowType.Int(32, true), null),
+ null)));
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+ allocator = new RootAllocator(Integer.MAX_VALUE);
+
+ final Location serverLocation = Location.forGrpcInsecure(FlightTestUtil.LOCALHOST, 0);
+ server = FlightServer.builder(allocator, serverLocation, new FlightSqlExample(serverLocation)).build();
+ server.start();
+
+ final Location clientLocation = Location.forGrpcInsecure(FlightTestUtil.LOCALHOST, server.getPort());
+ client = FlightClient.builder(allocator, clientLocation).build();
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ AutoCloseables.close(client, server, allocator);
+ }
+
+ /*
+ @Test
+ public void testGetTables() throws Exception {
+ // Arrange
+ final ActionGetTablesResult expected = ActionGetTablesResult.newBuilder()
+ .setSchema("APP")
+ .setTable("INTTABLE")
+ .setTableType("TABLE")
+ .setArrowMetadata(ByteString.copyFrom(SCHEMA_INT_TABLE.toByteArray()))
+ .build();
+
+ // Act
+ final Iterator<Result> results = client.doAction(new Action("GetTables",
+ Any.pack(ActionGetTablesRequest
+ .newBuilder()
+ .addTableTypes("TABLE")
+ .setIncludeSchema(true)
+ .build())
+ .toByteArray()));
+
+ // Assert
+ while (results.hasNext()) {
+ ActionGetTablesResult actual = Any.parseFrom(results.next().getBody()).unpack(ActionGetTablesResult.class);
+ assertEquals(expected, actual);
+ }
+ }
+
+ @Test
+ public void testGetTablesWithFlightSqlClientUtils() throws Exception {
+ // Arrange
+ final ActionGetTablesResult expected = ActionGetTablesResult.newBuilder()
+ .setSchema("APP")
+ .setTable("INTTABLE")
+ .setTableType("TABLE")
+ .setArrowMetadata(ByteString.copyFrom(SCHEMA_INT_TABLE.toByteArray()))
+ .build();
+
+ // Act
+ final List<ActionGetTablesResult> results = getTables(client, null, null, null,
+ Collections.singletonList("TABLE"), true);
+
+ // Assert
+ assertEquals(1, results.size());
+ assertEquals(expected, results.get(0));
+ }
+ */
+
+ @Test
+ public void testSimplePrepStmt() throws Exception {
+ final Iterator<Result> preparedStatementResults = client.doAction(new Action("GetPreparedStatement",
+ Any.pack(ActionCreatePreparedStatementRequest
+ .newBuilder()
+ .setQuery("Select * from intTable")
+ .build())
+ .toByteArray()));
+
+ assertTrue(preparedStatementResults.hasNext());
+ final ActionCreatePreparedStatementResult preparedStatementResult =
+ Any.parseFrom(preparedStatementResults.next().getBody()).unpack(ActionCreatePreparedStatementResult.class);
+ assertFalse(preparedStatementResults.hasNext());
+
+ final Schema actualSchema = Schema.deserialize(preparedStatementResult.getDatasetSchema().asReadOnlyByteBuffer());
+ assertEquals(SCHEMA_INT_TABLE, actualSchema);
+
+ final FlightDescriptor descriptor = FlightDescriptor
+ .command(Any.pack(CommandPreparedStatementQuery.newBuilder()
+ .setClientExecutionHandle(ByteString.copyFrom(new byte[]{1, 2, 3, 4}))
+ .setPreparedStatementHandle(preparedStatementResult.getPreparedStatementHandle())
+ .build())
+ .toByteArray());
+
+ final FlightInfo info = client.getInfo(descriptor);
+ assertEquals(SCHEMA_INT_TABLE, info.getSchema());
+
+ final FlightStream stream = client.getStream(info.getEndpoints().get(0).getTicket());
Review comment:
Please fix throughout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664915445
##########
File path: java/flight/flight-sql/src/test/java/org/apache/arrow/flight/TestFlightSql.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight;
+
+import static org.apache.arrow.flight.sql.FlightSqlClientUtils.getPreparedStatement;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.arrow.flight.sql.FlightSqlClientUtils;
+import org.apache.arrow.flight.sql.FlightSqlExample;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionClosePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionCreatePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionCreatePreparedStatementResult;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementQuery;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.memory.util.ArrowBufPointer;
+import org.apache.arrow.util.AutoCloseables;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.arrow.vector.util.ElementAddressableVectorIterator;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.ByteString;
+
+/**
+ * Test direct usage of Flight SQL workflows.
+ */
+public class TestFlightSql {
+ private static BufferAllocator allocator;
+ private static FlightServer server;
+
+ private static FlightClient client;
+
+ protected static final Schema SCHEMA_INT_TABLE = new Schema(Arrays.asList(
+ new Field("KEYNAME", new
+ FieldType(true, ArrowType.Utf8.INSTANCE, null),
+ null),
+ new Field("VALUE",
+ new FieldType(true, new ArrowType.Int(32, true), null),
+ null)));
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+ allocator = new RootAllocator(Integer.MAX_VALUE);
+
+ final Location serverLocation = Location.forGrpcInsecure(FlightTestUtil.LOCALHOST, 0);
+ server = FlightServer.builder(allocator, serverLocation, new FlightSqlExample(serverLocation)).build();
+ server.start();
+
+ final Location clientLocation = Location.forGrpcInsecure(FlightTestUtil.LOCALHOST, server.getPort());
+ client = FlightClient.builder(allocator, clientLocation).build();
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ AutoCloseables.close(client, server, allocator);
+ }
+
+ /*
+ @Test
+ public void testGetTables() throws Exception {
+ // Arrange
+ final ActionGetTablesResult expected = ActionGetTablesResult.newBuilder()
+ .setSchema("APP")
+ .setTable("INTTABLE")
+ .setTableType("TABLE")
+ .setArrowMetadata(ByteString.copyFrom(SCHEMA_INT_TABLE.toByteArray()))
+ .build();
+
+ // Act
+ final Iterator<Result> results = client.doAction(new Action("GetTables",
+ Any.pack(ActionGetTablesRequest
+ .newBuilder()
+ .addTableTypes("TABLE")
+ .setIncludeSchema(true)
+ .build())
+ .toByteArray()));
+
+ // Assert
+ while (results.hasNext()) {
+ ActionGetTablesResult actual = Any.parseFrom(results.next().getBody()).unpack(ActionGetTablesResult.class);
+ assertEquals(expected, actual);
+ }
+ }
+
+ @Test
+ public void testGetTablesWithFlightSqlClientUtils() throws Exception {
+ // Arrange
+ final ActionGetTablesResult expected = ActionGetTablesResult.newBuilder()
+ .setSchema("APP")
+ .setTable("INTTABLE")
+ .setTableType("TABLE")
+ .setArrowMetadata(ByteString.copyFrom(SCHEMA_INT_TABLE.toByteArray()))
+ .build();
+
+ // Act
+ final List<ActionGetTablesResult> results = getTables(client, null, null, null,
+ Collections.singletonList("TABLE"), true);
+
+ // Assert
+ assertEquals(1, results.size());
+ assertEquals(expected, results.get(0));
+ }
+ */
+
+ @Test
+ public void testSimplePrepStmt() throws Exception {
+ final Iterator<Result> preparedStatementResults = client.doAction(new Action("GetPreparedStatement",
+ Any.pack(ActionCreatePreparedStatementRequest
+ .newBuilder()
+ .setQuery("Select * from intTable")
+ .build())
+ .toByteArray()));
+
+ assertTrue(preparedStatementResults.hasNext());
+ final ActionCreatePreparedStatementResult preparedStatementResult =
+ Any.parseFrom(preparedStatementResults.next().getBody()).unpack(ActionCreatePreparedStatementResult.class);
+ assertFalse(preparedStatementResults.hasNext());
+
+ final Schema actualSchema = Schema.deserialize(preparedStatementResult.getDatasetSchema().asReadOnlyByteBuffer());
+ assertEquals(SCHEMA_INT_TABLE, actualSchema);
+
+ final FlightDescriptor descriptor = FlightDescriptor
+ .command(Any.pack(CommandPreparedStatementQuery.newBuilder()
+ .setClientExecutionHandle(ByteString.copyFrom(new byte[]{1, 2, 3, 4}))
+ .setPreparedStatementHandle(preparedStatementResult.getPreparedStatementHandle())
+ .build())
+ .toByteArray());
+
+ final FlightInfo info = client.getInfo(descriptor);
+ assertEquals(SCHEMA_INT_TABLE, info.getSchema());
+
+ final FlightStream stream = client.getStream(info.getEndpoints().get(0).getTicket());
+ assertEquals(SCHEMA_INT_TABLE, stream.getSchema());
+
+ List<String> actualStringResults = new ArrayList<>();
+ List<Integer> actualIntResults = new ArrayList<>();
+ while (stream.next()) {
+ final VectorSchemaRoot root = stream.getRoot();
+ final long rowCount = root.getRowCount();
+
+ for (Field field : root.getSchema().getFields()) {
+ final FieldVector fieldVector = root.getVector(field.getName());
+
+ if (fieldVector instanceof VarCharVector) {
+
+ final ElementAddressableVectorIterator<VarCharVector> it =
+ new ElementAddressableVectorIterator<>((VarCharVector) fieldVector);
+
+ for (int rowIndex = 0; rowIndex < rowCount; rowIndex++) {
+ final ArrowBufPointer pt = it.next();
+ final byte[] bytes = new byte[(int) pt.getLength()];
+ pt.getBuf().getBytes(pt.getOffset(), bytes);
+
+ actualStringResults.add(new String(bytes, StandardCharsets.UTF_8));
+ }
+ } else if (fieldVector instanceof IntVector) {
+ for (int rowIndex = 0; rowIndex < rowCount; rowIndex++) {
+ actualIntResults.add(((IntVector) fieldVector).get(rowIndex));
+ }
+ }
+ }
+ }
+ stream.getRoot().clear();
+
+ assertEquals(Arrays.asList("one", "zero", "negative one"), actualStringResults);
+ assertEquals(Arrays.asList(1, 0, -1), actualIntResults);
+
+ final Iterator<Result> closePreparedStatementResults = client.doAction(new Action("ClosePreparedStatement",
+ Any.pack(ActionClosePreparedStatementRequest
+ .newBuilder()
+ .setPreparedStatementHandleBytes(preparedStatementResult.getPreparedStatementHandle())
+ .build())
+ .toByteArray()));
+ assertFalse(closePreparedStatementResults.hasNext());
+ }
+
+ @Test
+ public void testSimplePrepStmtWithFlightSqlClientUtils() throws Exception {
+ final FlightSqlClientUtils.FlightSqlPreparedStatement preparedStatement =
+ getPreparedStatement(client, "Select * from intTable");
+
+ final Schema actualSchema = preparedStatement.getResultSetSchema();
+ assertEquals(SCHEMA_INT_TABLE, actualSchema);
+
+ final FlightInfo info = preparedStatement.executeQuery();
+ assertEquals(SCHEMA_INT_TABLE, info.getSchema());
+
+ final FlightStream stream = client.getStream(info.getEndpoints().get(0).getTicket());
+ assertEquals(SCHEMA_INT_TABLE, stream.getSchema());
+
+ List<String> actualStringResults = new ArrayList<>();
+ List<Integer> actualIntResults = new ArrayList<>();
+ while (stream.next()) {
+ final VectorSchemaRoot root = stream.getRoot();
+ final long rowCount = root.getRowCount();
+
+ for (Field field : root.getSchema().getFields()) {
+ final FieldVector fieldVector = root.getVector(field.getName());
+
+ if (fieldVector instanceof VarCharVector) {
+
+ final ElementAddressableVectorIterator<VarCharVector> it =
+ new ElementAddressableVectorIterator<>((VarCharVector) fieldVector);
+
+ for (int rowIndex = 0; rowIndex < rowCount; rowIndex++) {
+ final ArrowBufPointer pt = it.next();
+ final byte[] bytes = new byte[(int) pt.getLength()];
+ pt.getBuf().getBytes(pt.getOffset(), bytes);
+
+ actualStringResults.add(new String(bytes, StandardCharsets.UTF_8));
+ }
+ } else if (fieldVector instanceof IntVector) {
+ for (int rowIndex = 0; rowIndex < rowCount; rowIndex++) {
+ actualIntResults.add(((IntVector) fieldVector).get(rowIndex));
+ }
+ }
+ }
+ }
+ stream.getRoot().clear();
+
+ assertEquals(Arrays.asList("one", "zero", "negative one"), actualStringResults);
+ assertEquals(Arrays.asList(1, 0, -1), actualIntResults);
+
+ AutoCloseables.close(preparedStatement);
Review comment:
try-with-resources instead of explicit close.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r661432341
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
+
+ // Implementation specific custom SqlInfo values.
+ repeated string custom_info = 2;
+}
+
+/*
+ Represents a request to retrieve the list of catalogs on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name.
+ */
+message CommandGetCatalogs {
+}
+
+/*
+ * Represents a request to retrieve the list of schemas on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8
+ * >
Review comment:
Let's keep this then.
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
Review comment:
Sounds good. Is there still a need for `custom_info` then?
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
+
+ // Implementation specific custom SqlInfo values.
+ repeated string custom_info = 2;
+}
+
+/*
+ Represents a request to retrieve the list of catalogs on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name.
+ */
+message CommandGetCatalogs {
+}
+
+/*
+ * Represents a request to retrieve the list of schemas on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name, then schema_name.
+ */
+message CommandGetSchemas {
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then all catalogs are searched.
+ */
+ string catalog = 1;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 2;
+}
+
+/*
+ * Represents a request to retrieve the list of tables, and optionally their schemas, on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8,
+ * table_name: utf8,
+ * table_type: utf8,
+ * table_schema: bytes
+ * >
+ * The returned data should be ordered by catalog_name, schema_name, table_name, then table_type.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then all catalogs are searched.
+ */
+ string catalog = 1;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, all schemas matching other filters are searched.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 2;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no table_name_filter_pattern is provided, all tables matching other filters are searched.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 3;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 4;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 5;
+}
+
+/*
+ * Represents a request to retrieve the list of table types on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * table_type: utf8
+ * >
+ * The returned data should be ordered by table_type.
+ */
+message CommandGetTableTypes {
+}
Review comment:
Ah, right, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r652027498
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
Review comment:
In many cases retrieval of the schema of a table is non-trivial, but BI tools often list many tables before getting detailed metadata about one or two of the tables. This is the reason for the breakdown in ODBC/JDBC of getTables vs. getColumns. I think we should break the schemas for a table out to a separate pseudo table for efficiency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r652300305
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 6;
+}
+
+/*
+ * Wrap the result of a "GetTables" action.
+ */
+message ActionGetTablesResult {
+ string catalog = 1;
+ string schema = 2;
+ string table = 3;
+ string table_type = 4;
+
+ /*
+ * Schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ * Null if include_schema on request is false.
+ */
+ bytes arrow_metadata = 5;
+}
+
+/*
+ * Wrap the result of a "GetTableTypes" action.
+ */
+message ActionGetTableTypesResult {
+ /*
+ * Indicates the type of the table. E.g. table (regular data table) , view, system table etc.
+ */
+ string table_type = 1;
+}
+
+// SQL Execution Action Messages
+
+/*
+ * Request message for the "GetPreparedStatement" action on a
+ * Flight SQL enabled backend.
+ */
+message ActionGetPreparedStatementRequest {
+ // The valid SQL string to get a prepared statement for.
+ string query = 1;
+}
+
+/*
+ * Wrap the result of a "GetPreparedStatement" action.
+ */
+message ActionGetPreparedStatementResult {
+
+ // Opaque handle for the prepared statement on the server.
+ bytes prepared_statement_handle = 1;
+
+ // If a result set generating query was provided, dataset_schema contains the
+ // schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ bytes dataset_schema = 2;
+
+ // If the query provided contained parameters, parameter_schema contains the
+ // Schema of the expected parameters as described in Schema.fbs::Schema.
+ bytes parameter_schema = 3;
+}
+
+/*
+ * Request message for the "ClosePreparedStatement" action on a
+ * Flight SQL enabled backend.
+ * Closes server resources associated with the prepared statement handle.
+ */
+message ActionClosePreparedStatementRequest {
+ // Opaque handle for the prepared statement on the server.
+ string prepared_statement_handle = 1;
+}
+
+
+// SQL Execution Messages.
+
+/*
+ * Represents a SQL query. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ */
+message CommandStatementQuery {
+ // The SQL syntax.
+ string query = 1;
+}
+
+/*
+ * Represents an instance of executing a prepared statement. Used in the
+ * command member of FlightDescriptor for the following RPC calls:
+ * TODO: Is the idea that a Put with parameter values would execute multiple bound versions of the prepared statement
Review comment:
@jduo and I were discussing, and realized that there's no allowance for output or I/O parameters. Do you happen to have any ideas about how we could integrate that into the flow without breaking anything too much? I'd hate to have to use separate calls to get the parameter data back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-770461359
Thank you for the PR. I think since there isn't a fully implementation example yet for the all the features it would make this easier to review if we cut down the FlightSql proto to what is implemented and focus on each use-case in a separate PR. I think it makes sense to handle read, write and metadata operations separately?
> Like other Flight APIs, flight-sql does not provide implementation details that dictate how a client and server communicates with each other, it simply provides the SQL semantics and apply them onto the Flight API.
IMO, I think we should be be trying to specify more about how client and server interact with each other in terms of error handling and retries, these are important aspects and without it I think some of the utility of standardization will be diminished.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r635407696
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
Review comment:
per comments elsewhere. I don't think we should be introduce a new type system here. Lets please use the Arrow type system for returning metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r567491286
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 6;
+}
+
+/*
+ * Wrap the result of a "GetTables" action.
+ */
+message ActionGetTablesResult {
+ string catalog = 1;
+ string schema = 2;
+ string table = 3;
+ string table_type = 4;
+
+ /*
+ * Schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ * Null if include_schema on request is false.
+ */
+ bytes arrow_metadata = 5;
+}
+
+/*
+ * Wrap the result of a "GetTableTypes" action.
+ */
+message ActionGetTableTypesResult {
+ /*
+ * Indicates the type of the table. E.g. table (regular data table) , view, system table etc.
Review comment:
```suggestion
* Indicates the type of the table. E.g. table (regular data table), view, system table etc.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-868694053
@lidavidm - that would be the goal. Am working to get this finalized, restrictions are just on time that can be made available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664913775
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSqlUtils.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.sql.Types;
+import java.util.List;
+
+import org.apache.arrow.flight.ActionType;
+import org.apache.arrow.vector.types.DateUnit;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+
+import com.google.common.collect.ImmutableList;
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+import com.google.protobuf.Message;
+
+/**
+ * Utilities to work with Flight SQL semantics.
+ */
+public final class FlightSqlUtils {
+
+ private static final int BIT_WIDTH8 = 8;
+ private static final int BIT_WIDTH_16 = 16;
+ private static final int BIT_WIDTH_32 = 32;
+ private static final int BIT_WIDTH_64 = 64;
+ private static final boolean IS_SIGNED_FALSE = false;
+ private static final boolean IS_SIGNED_TRUE = true;
+
+ public static final ActionType FLIGHT_SQL_CREATEPREPAREDSTATEMENT = new ActionType("CreatePreparedStatement",
+ "Creates a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionCreatePreparedStatementRequest\n" +
+ "Response Message: ActionCreatePreparedStatementResult");
+
+ public static final ActionType FLIGHT_SQL_CLOSEPREPAREDSTATEMENT = new ActionType("ClosePreparedStatement",
+ "Closes a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionClosePreparedStatementRequest\n" +
+ "Response Message: N/A");
+
+ public static final List<ActionType> FLIGHT_SQL_ACTIONS = ImmutableList.of(
+ FLIGHT_SQL_CREATEPREPAREDSTATEMENT,
+ FLIGHT_SQL_CLOSEPREPAREDSTATEMENT
+ );
+
+ /**
+ * Converts {@link java.sql.Types} values returned from JDBC Apis to Arrow types.
+ *
+ * @param jdbcDataType {@link java.sql.Types} value.
+ * @param precision Precision of the type.
+ * @param scale Scale of the type.
+ * @return The Arrow equivalent type.
+ */
+ public static ArrowType getArrowTypeFromJDBCType(int jdbcDataType, int precision, int scale) {
+
+ switch (jdbcDataType) {
+ case Types.BIT:
+ case Types.BOOLEAN:
+ return ArrowType.Bool.INSTANCE;
+ case Types.TINYINT:
+ return new ArrowType.Int(BIT_WIDTH8, IS_SIGNED_TRUE);
+ case Types.SMALLINT:
+ return new ArrowType.Int(BIT_WIDTH_16, IS_SIGNED_TRUE);
+ case Types.INTEGER:
+ return new ArrowType.Int(BIT_WIDTH_32, IS_SIGNED_TRUE);
+ case Types.BIGINT:
+ return new ArrowType.Int(BIT_WIDTH_64, IS_SIGNED_TRUE);
+ case Types.FLOAT:
+ case Types.REAL:
+ return new ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE);
+ case Types.DOUBLE:
+ return new ArrowType.FloatingPoint(FloatingPointPrecision.DOUBLE);
+ case Types.NUMERIC:
+ case Types.DECIMAL:
+ return new ArrowType.Decimal(precision, scale);
+ case Types.DATE:
+ return new ArrowType.Date(DateUnit.DAY);
+ case Types.TIME:
+ return new ArrowType.Time(TimeUnit.MILLISECOND, BIT_WIDTH_32);
+ case Types.TIMESTAMP:
+ return new ArrowType.Timestamp(TimeUnit.MILLISECOND, null);
+ case Types.BINARY:
+ case Types.VARBINARY:
+ case Types.LONGVARBINARY:
+ return ArrowType.Binary.INSTANCE;
+ case Types.NULL:
+ return ArrowType.Null.INSTANCE;
+
+ case Types.CHAR:
+ case Types.VARCHAR:
+ case Types.LONGVARCHAR:
+ case Types.CLOB:
+ case Types.NCHAR:
+ case Types.NVARCHAR:
+ case Types.LONGNVARCHAR:
+ case Types.NCLOB:
+
+ case Types.OTHER:
+ case Types.JAVA_OBJECT:
+ case Types.DISTINCT:
+ case Types.STRUCT:
+ case Types.ARRAY:
+ case Types.BLOB:
+ case Types.REF:
+ case Types.DATALINK:
+ case Types.ROWID:
+ case Types.SQLXML:
+ case Types.REF_CURSOR:
+ case Types.TIME_WITH_TIMEZONE:
Review comment:
Should be an Arrow TimetampTz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r650643672
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 6;
+}
+
+/*
+ * Wrap the result of a "GetTables" action.
+ */
+message ActionGetTablesResult {
+ string catalog = 1;
+ string schema = 2;
+ string table = 3;
+ string table_type = 4;
+
+ /*
+ * Schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ * Null if include_schema on request is false.
+ */
+ bytes arrow_metadata = 5;
+}
+
+/*
+ * Wrap the result of a "GetTableTypes" action.
+ */
+message ActionGetTableTypesResult {
+ /*
+ * Indicates the type of the table. E.g. table (regular data table) , view, system table etc.
+ */
+ string table_type = 1;
+}
+
+// SQL Execution Action Messages
+
+/*
+ * Request message for the "GetPreparedStatement" action on a
+ * Flight SQL enabled backend.
+ */
+message ActionGetPreparedStatementRequest {
+ // The valid SQL string to get a prepared statement for.
+ string query = 1;
+}
+
+/*
+ * Wrap the result of a "GetPreparedStatement" action.
+ */
+message ActionGetPreparedStatementResult {
+
+ // Opaque handle for the prepared statement on the server.
+ bytes prepared_statement_handle = 1;
+
+ // If a result set generating query was provided, dataset_schema contains the
+ // schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ bytes dataset_schema = 2;
+
+ // If the query provided contained parameters, parameter_schema contains the
+ // Schema of the expected parameters as described in Schema.fbs::Schema.
+ bytes parameter_schema = 3;
+}
+
+/*
+ * Request message for the "ClosePreparedStatement" action on a
+ * Flight SQL enabled backend.
+ * Closes server resources associated with the prepared statement handle.
+ */
+message ActionClosePreparedStatementRequest {
+ // Opaque handle for the prepared statement on the server.
+ string prepared_statement_handle = 1;
+}
+
+
+// SQL Execution Messages.
+
+/*
+ * Represents a SQL query. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ */
+message CommandStatementQuery {
+ // The SQL syntax.
+ string query = 1;
+}
+
+/*
+ * Represents an instance of executing a prepared statement. Used in the
+ * command member of FlightDescriptor for the following RPC calls:
+ * TODO: Is the idea that a Put with parameter values would execute multiple bound versions of the prepared statement
+ * TODO: (one for each row)? Seems like that will work ok for Insert statements, but what about other kinds of prepared statements?
+ * - DoPut: bind parameter values.
+ * - GetFlightInfo: execute the prepared statement instance.
+ */
+message CommandPreparedStatementQuery {
+ // Unique identifier for the instance of the prepared statement to execute.
+ bytes client_execution_handle = 1;
+ // Opaque handle for the prepared statement on the server.
+ bytes prepared_statement_handle = 2;
+}
+
+/*
+ * Represents a SQL update query. Used in the command member of FlightDescriptor
+ * for the the RPC call DoPut to cause the server to execute the included
+ * SQL update.
+ */
+message CommandStatementUpdate {
+ // The SQL syntax.
+ string query = 1;
+}
+
+/*
+ * Represents a SQL update query. Used in the command member of FlightDescriptor
+ * for the the RPC call DoPut to cause the server to execute the included
+ * prepared statement handle as an update.
+ */
+message CommandPreparedStatementUpdate {
+ // Unique identifier for the instance of the prepared statement to execute.
+ bytes client_execution_handle = 1;
+ // Opaque handle for the prepared statement on the server.
+ bytes prepared_statement_handle = 2;
+}
+
+/*
+ * Returned from the RPC call DoPut when a CommandStatementUpdate
+ * CommandPreparedStatementUpdate was in the request, containing
+ * results from the update.
+ */
+message DoPutUpdateResult {
+ // The number of records updated. A return value of -1 represents
+ // an unknown updated record count.
+ int64 record_count = 1;
+}
+
+message ResultsOrder {
Review comment:
I think this concept should be removed entirely - under what circumstances would you want to specify an order? It seems cleaner to specify an order and then allow filtering to specifics.
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 6;
+}
+
+/*
+ * Wrap the result of a "GetTables" action.
+ */
+message ActionGetTablesResult {
+ string catalog = 1;
+ string schema = 2;
+ string table = 3;
+ string table_type = 4;
+
+ /*
+ * Schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ * Null if include_schema on request is false.
+ */
+ bytes arrow_metadata = 5;
+}
+
+/*
+ * Wrap the result of a "GetTableTypes" action.
+ */
+message ActionGetTableTypesResult {
+ /*
+ * Indicates the type of the table. E.g. table (regular data table) , view, system table etc.
+ */
+ string table_type = 1;
+}
+
+// SQL Execution Action Messages
+
+/*
+ * Request message for the "GetPreparedStatement" action on a
+ * Flight SQL enabled backend.
+ */
+message ActionGetPreparedStatementRequest {
Review comment:
I find the naming of GetPreparedStatement for creating the statement, and PreparedStatement for execution confusing.
Would suggest instead:
- ActionCreatePreparedStatementRequest
- ActionExecutePreparedStatementRequest
- ActionClosePreparedStatementRequest
instead, although perhaps that's just me.
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 6;
+}
+
+/*
+ * Wrap the result of a "GetTables" action.
+ */
+message ActionGetTablesResult {
+ string catalog = 1;
+ string schema = 2;
+ string table = 3;
+ string table_type = 4;
+
+ /*
+ * Schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ * Null if include_schema on request is false.
+ */
+ bytes arrow_metadata = 5;
+}
+
+/*
+ * Wrap the result of a "GetTableTypes" action.
+ */
+message ActionGetTableTypesResult {
+ /*
+ * Indicates the type of the table. E.g. table (regular data table) , view, system table etc.
+ */
+ string table_type = 1;
+}
+
+// SQL Execution Action Messages
+
+/*
+ * Request message for the "GetPreparedStatement" action on a
+ * Flight SQL enabled backend.
+ */
+message ActionGetPreparedStatementRequest {
Review comment:
I find the naming of GetPreparedStatement for creating the statement, and PreparedStatement for execution confusing.
Would suggest instead:
- ActionCreatePreparedStatementRequest
- CommandExecutePreparedStatementRequest
- ActionClosePreparedStatementRequest
instead, although perhaps that's just me.
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 6;
+}
+
+/*
+ * Wrap the result of a "GetTables" action.
+ */
+message ActionGetTablesResult {
+ string catalog = 1;
+ string schema = 2;
+ string table = 3;
+ string table_type = 4;
+
+ /*
+ * Schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ * Null if include_schema on request is false.
+ */
+ bytes arrow_metadata = 5;
+}
+
+/*
+ * Wrap the result of a "GetTableTypes" action.
+ */
+message ActionGetTableTypesResult {
+ /*
+ * Indicates the type of the table. E.g. table (regular data table) , view, system table etc.
+ */
+ string table_type = 1;
+}
+
+// SQL Execution Action Messages
+
+/*
+ * Request message for the "GetPreparedStatement" action on a
+ * Flight SQL enabled backend.
+ */
+message ActionGetPreparedStatementRequest {
Review comment:
I find the naming of GetPreparedStatement for creating the statement, and PreparedStatement for execution confusing.
Would suggest instead:
- ActionCreatePreparedStatementRequest
- CommandExecutePreparedStatement
- ActionClosePreparedStatementRequest
instead, although perhaps that's just me.
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 6;
+}
+
+/*
+ * Wrap the result of a "GetTables" action.
+ */
+message ActionGetTablesResult {
+ string catalog = 1;
+ string schema = 2;
+ string table = 3;
+ string table_type = 4;
+
+ /*
+ * Schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ * Null if include_schema on request is false.
+ */
+ bytes arrow_metadata = 5;
+}
+
+/*
+ * Wrap the result of a "GetTableTypes" action.
+ */
+message ActionGetTableTypesResult {
+ /*
+ * Indicates the type of the table. E.g. table (regular data table) , view, system table etc.
+ */
+ string table_type = 1;
+}
+
+// SQL Execution Action Messages
+
+/*
+ * Request message for the "GetPreparedStatement" action on a
+ * Flight SQL enabled backend.
+ */
+message ActionGetPreparedStatementRequest {
Review comment:
I find the naming of GetPreparedStatement for creating the statement, and PreparedStatement for execution confusing.
Would suggest instead:
- ActionCreatePreparedStatementRequest
- ActionClosePreparedStatementRequest
- CommandExecutePreparedStatementResult
- CommandExecutePreparedStatementUpdate (or Rowcount)
instead, although perhaps that's just me.
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 6;
+}
+
+/*
+ * Wrap the result of a "GetTables" action.
+ */
+message ActionGetTablesResult {
+ string catalog = 1;
+ string schema = 2;
+ string table = 3;
+ string table_type = 4;
+
+ /*
+ * Schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ * Null if include_schema on request is false.
+ */
+ bytes arrow_metadata = 5;
+}
+
+/*
+ * Wrap the result of a "GetTableTypes" action.
+ */
+message ActionGetTableTypesResult {
+ /*
+ * Indicates the type of the table. E.g. table (regular data table) , view, system table etc.
+ */
+ string table_type = 1;
+}
+
+// SQL Execution Action Messages
+
+/*
+ * Request message for the "GetPreparedStatement" action on a
+ * Flight SQL enabled backend.
+ */
+message ActionGetPreparedStatementRequest {
Review comment:
I find the naming of GetPreparedStatement for creating the statement, and PreparedStatement for execution confusing.
Would suggest instead:
- ActionCreatePreparedStatementRequest
- ActionClosePreparedStatementRequest
- CommandExecutePreparedStatementQuery (or Result, which I think is clearer)
- CommandExecutePreparedStatementUpdate (or Rowcount)
instead, although perhaps that's just me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-871555513
@lidavidm - thanks for the reminder, I'll do that since I've already fixed the casing everywhere else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r567492525
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSQLUtils.java
##########
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.sql.Types;
+import java.util.List;
+
+import org.apache.arrow.flight.ActionType;
+import org.apache.arrow.vector.types.DateUnit;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+
+import com.google.common.collect.ImmutableList;
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+import com.google.protobuf.Message;
+
+/**
+ * Utilities to work with Flight SQL semantics.
+ */
+public final class FlightSQLUtils {
+
+ private static final int BIT_WIDTH8 = 8;
+ private static final int BIT_WIDTH_16 = 16;
+ private static final int BIT_WIDTH_32 = 32;
+ private static final int BIT_WIDTH_64 = 64;
+ private static final boolean IS_SIGNED_FALSE = false;
+ private static final boolean IS_SIGNED_TRUE = true;
+
+ public static final ActionType FLIGHT_SQL_GETSQLINFO = new ActionType("GetSQLINFO",
+ "Retrieves details of SQL capabilities of the Flight server. \n" +
+ "Request Message: N/A\n" +
+ "Response Message: ActionGetSQLInfoResult");
+
+ public static final ActionType FLIGHT_SQL_GETCATALOGS = new ActionType("GetCatalogs",
+ "Retrieves a list of all catalogs available on the server. \n" +
+ "Request Message: ActionGetCatalogsRequest\n" +
+ "Response Message: ActionGetCatalogsResult");
+
+ public static final ActionType FLIGHT_SQL_GETSCHEMAS = new ActionType("GetSchemas",
+ "Retrieves a list of schemas available on the server. \n" +
+ "Request Message: ActionGetSchemasRequest\n" +
+ "Response Message: ActionGetSchemasResult");
+
+ public static final ActionType FLIGHT_SQL_GETTABLES = new ActionType("GetTables",
+ "Retrieves a list of tables available on the server. \n" +
+ "Request Message: ActionGetTablesRequest\n" +
+ "Response Message: ActionGetTablesResult");
+
+ public static final ActionType FLIGHT_SQL_GETTABLETYPES = new ActionType("GetTableTypes",
+ "Retrieves a list of table types available on the server. \n" +
+ "Request Message: N/A\n" +
+ "Response Message: ActionGetTableTypesResult");
+
+ public static final ActionType FLIGHT_SQL_GETPREPAREDSTATEMENT = new ActionType("GetPreparedStatement",
+ "Creates a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionGetPreparedStatementRequest\n" +
+ "Response Message: ActionGetPreparedStatementResult");
+
+ public static final ActionType FLIGHT_SQL_CLOSEPREPAREDSTATEMENT = new ActionType("ClosePreparedStatement",
+ "Closes a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionClosePreparedStatementRequest\n" +
+ "Response Message: N/A");
+
+ public static final List<ActionType> FLIGHT_SQL_ACTIONS = ImmutableList.of(
+ FLIGHT_SQL_GETSQLINFO,
+ FLIGHT_SQL_GETCATALOGS,
+ FLIGHT_SQL_GETSCHEMAS,
+ FLIGHT_SQL_GETTABLES,
+ FLIGHT_SQL_GETTABLETYPES,
+ FLIGHT_SQL_GETPREPAREDSTATEMENT,
+ FLIGHT_SQL_CLOSEPREPAREDSTATEMENT
+ );
+
+ /**
+ * Converts {@link java.sql.Types} values returned from JDBC Apis to Arrow types.
+ *
+ * @param jdbcDataType {@link java.sql.Types} value.
+ * @param precision Precision of the type.
+ * @param scale Scale of the type.
+ * @return The Arrow equivalent type.
+ */
+ public static ArrowType getArrowTypeFromJDBCType(int jdbcDataType, int precision, int scale) {
Review comment:
this duplicates code in the jdbc to arrow package i believe, can we consolidate it? I also thing there might be the need for custom mappings (we needed it JDBC conversion)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r658923471
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 6;
+}
+
+/*
+ * Wrap the result of a "GetTables" action.
+ */
+message ActionGetTablesResult {
+ string catalog = 1;
+ string schema = 2;
+ string table = 3;
+ string table_type = 4;
+
+ /*
+ * Schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ * Null if include_schema on request is false.
+ */
+ bytes arrow_metadata = 5;
+}
+
+/*
+ * Wrap the result of a "GetTableTypes" action.
+ */
+message ActionGetTableTypesResult {
+ /*
+ * Indicates the type of the table. E.g. table (regular data table) , view, system table etc.
+ */
+ string table_type = 1;
+}
+
+// SQL Execution Action Messages
+
+/*
+ * Request message for the "GetPreparedStatement" action on a
+ * Flight SQL enabled backend.
+ */
+message ActionGetPreparedStatementRequest {
+ // The valid SQL string to get a prepared statement for.
+ string query = 1;
+}
+
+/*
+ * Wrap the result of a "GetPreparedStatement" action.
+ */
+message ActionGetPreparedStatementResult {
+
+ // Opaque handle for the prepared statement on the server.
+ bytes prepared_statement_handle = 1;
+
+ // If a result set generating query was provided, dataset_schema contains the
+ // schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ bytes dataset_schema = 2;
+
+ // If the query provided contained parameters, parameter_schema contains the
+ // Schema of the expected parameters as described in Schema.fbs::Schema.
+ bytes parameter_schema = 3;
+}
+
+/*
+ * Request message for the "ClosePreparedStatement" action on a
+ * Flight SQL enabled backend.
+ * Closes server resources associated with the prepared statement handle.
+ */
+message ActionClosePreparedStatementRequest {
+ // Opaque handle for the prepared statement on the server.
+ string prepared_statement_handle = 1;
+}
+
+
+// SQL Execution Messages.
+
+/*
+ * Represents a SQL query. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ */
+message CommandStatementQuery {
+ // The SQL syntax.
+ string query = 1;
+}
+
+/*
+ * Represents an instance of executing a prepared statement. Used in the
+ * command member of FlightDescriptor for the following RPC calls:
+ * TODO: Is the idea that a Put with parameter values would execute multiple bound versions of the prepared statement
Review comment:
I think the initial approach should be to exclude output parameters and then rev the FlightSQL version afterwards to include them once the schema change proposal has been finalized, so we can get this moving forward. As mentioned in the weekly status call, there is a JDBC implementation being built to take advantage of this so we want to ensure everything moves forward together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r635405484
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSQLClientUtils.java
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.arrow.flight.Action;
+import org.apache.arrow.flight.FlightClient;
+import org.apache.arrow.flight.FlightDescriptor;
+import org.apache.arrow.flight.FlightInfo;
+import org.apache.arrow.flight.Result;
+import org.apache.arrow.flight.sql.impl.FlightSQL;
+import org.apache.arrow.flight.sql.impl.FlightSQL.ActionGetPreparedStatementResult;
+import org.apache.arrow.flight.sql.impl.FlightSQL.ActionGetTablesRequest;
+import org.apache.arrow.flight.sql.impl.FlightSQL.ActionGetTablesResult;
+import org.apache.arrow.flight.sql.impl.FlightSQL.CommandPreparedStatementQuery;
+import org.apache.arrow.vector.types.pojo.Schema;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.ByteString;
+
+import io.grpc.Status;
+
+/**
+ * Client side utilities to work with Flight SQL semantics.
+ */
+public final class FlightSQLClientUtils {
Review comment:
We also don't have a way to cancel one of these metadata requests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r660925732
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
+
+ // Implementation specific custom SqlInfo values.
+ repeated string custom_info = 2;
+}
+
+/*
+ Represents a request to retrieve the list of catalogs on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name.
+ */
+message CommandGetCatalogs {
+}
+
+/*
+ * Represents a request to retrieve the list of schemas on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8
+ * >
Review comment:
I can go either way, the data overhead for repetition can be non-trivial so I like the idea of making it something else, I chose this for simplicity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield edited a comment on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield edited a comment on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-776084026
> Thinking about this a bit more I think an alternative to a lot of the metadata querying is to define a minimal sql table schema and querying capabilities for it. I'll expand more in a little bit
I did a rough sketch on how the metadata layer could be modeled as pseudo-tables as comments on the proto. I think this addresses some uneasiner I had with the existing model:
1. It uses one consistent data model for tabular data (the arrow model).
2. It makes query semantics clearer instead of trying to model them in an ad-hoc manner through protobufs
This comes at slightly more complex for server implementations. And potential confusion on limitations for specific tables. This might be limited to some extend if we had a different GetMetadata action so implementations know to expect a query to specific pseudo-table. I expect we would provide library wrappers over JDBC to also ease implementation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r652299841
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
Review comment:
I think using pseudo tables is an interesting concept, with some interesting pros/cons worth discussing to decide on an approach.
Using pseudo tables allows for:
- a simple proto definition
- more expressiveness in terms of what can be done with the table, since it is then available for general SQL
- allowing later additions without needing to modify any proto
Drawbacks are mostly around two cases: those where metadata is not retrieved via a query mechanism and/or where it's more efficient not to use a query mechanism, and where you need to layer an additional parser in front of the actual database parser to support the pseudo. If we add a partial parser implementation to make it easier, we are having to implement this in all the languages that Flight SQL supports.
If we use the existing proposed proto definitions, then I see the following:
- less expressive in terms of what can be done
- less flexibility in terms of adding new metadata calls
- much less effort in that we don't need to provide a partial parser implementation to help
- It's easy to go from a message to a SQL query if that's what's used for metadata, it's hard to go the other way around
I think it would be easier all around to use what's defined here (with tweaks) vs. going the SQL table route, even though the SQL table route is more elegant from a protocol perspective. The effort to provide the hand-up for those data sources which don't have any support for querying metadata would be large, and may be a hinderance to adoption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r572929697
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
Review comment:
These two seem redundant.
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
Review comment:
Where does this syntax come from? (If it's a particular vendor, can we avoid it in favor of more standard glob or SQL LIKE expressions?)
##########
File path: java/flight/flight-sql/src/test/protobuf/flightSQLExample.proto
##########
@@ -0,0 +1,26 @@
+/*
Review comment:
Additionally shouldn't Protobuf file names be in snake_case? See https://developers.google.com/protocol-buffers/docs/style#file_structure
##########
File path: java/flight/flight-sql/src/test/protobuf/flightSQLExample.proto
##########
@@ -0,0 +1,26 @@
+/*
Review comment:
I guess Flight.proto itself is already an exception...we should be consistent with what we have, though, yes (so `FlightSQLExample.proto` or preferably `FlightSql.proto` and `FlightSqlExample.proto`)
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSQLUtils.java
##########
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.sql.Types;
+import java.util.List;
+
+import org.apache.arrow.flight.ActionType;
+import org.apache.arrow.vector.types.DateUnit;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+
+import com.google.common.collect.ImmutableList;
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+import com.google.protobuf.Message;
+
+/**
+ * Utilities to work with Flight SQL semantics.
+ */
+public final class FlightSQLUtils {
+
+ private static final int BIT_WIDTH8 = 8;
+ private static final int BIT_WIDTH_16 = 16;
+ private static final int BIT_WIDTH_32 = 32;
+ private static final int BIT_WIDTH_64 = 64;
+ private static final boolean IS_SIGNED_FALSE = false;
+ private static final boolean IS_SIGNED_TRUE = true;
+
+ public static final ActionType FLIGHT_SQL_GETSQLINFO = new ActionType("GetSQLINFO",
Review comment:
nit: GetSqlInfo?
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 6;
+}
+
+/*
+ * Wrap the result of a "GetTables" action.
+ */
+message ActionGetTablesResult {
+ string catalog = 1;
+ string schema = 2;
+ string table = 3;
+ string table_type = 4;
+
+ /*
+ * Schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ * Null if include_schema on request is false.
+ */
+ bytes arrow_metadata = 5;
+}
+
+/*
+ * Wrap the result of a "GetTableTypes" action.
+ */
+message ActionGetTableTypesResult {
+ /*
+ * Indicates the type of the table. E.g. table (regular data table) , view, system table etc.
+ */
+ string table_type = 1;
+}
+
+// SQL Execution Action Messages
+
+/*
+ * Request message for the "GetPreparedStatement" action on a
+ * Flight SQL enabled backend.
+ */
+message ActionGetPreparedStatementRequest {
+ // The valid SQL string to get a prepared statement for.
+ string query = 1;
+}
+
+/*
+ * Wrap the result of a "GetPreparedStatement" action.
+ */
+message ActionGetPreparedStatementResult {
+
+ // Opaque handle for the prepared statement on the server.
+ bytes prepared_statement_handle = 1;
+
+ // If a result set generating query was provided, dataset_schema contains the
+ // schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ bytes dataset_schema = 2;
+
+ // If the query provided contained parameters, parameter_schema contains the
+ // Schema of the expected parameters as described in Schema.fbs::Schema.
+ bytes parameter_schema = 3;
+}
+
+/*
+ * Request message for the "ClosePreparedStatement" action on a
+ * Flight SQL enabled backend.
+ * Closes server resources associated with the prepared statement handle.
+ */
+message ActionClosePreparedStatementRequest {
+ // Opaque handle for the prepared statement on the server.
+ string prepared_statement_handle = 1;
+}
+
+
+// SQL Execution Messages.
+
+/*
+ * Represents a SQL query. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ */
+message CommandStatementQuery {
+ // The SQL syntax.
+ string query = 1;
+}
+
+/*
+ * Represents an instance of executing a prepared statement. Used in the
+ * command member of FlightDescriptor for the following RPC calls:
+ * TODO: Is the idea that a Put with parameter values would execute multiple bound versions of the prepared statement
Review comment:
Just a thought: you could use DoExchange to combine the steps of providing parameter values and retrieving the results.
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
Review comment:
Library version? Or format version?
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSQLProducer.java
##########
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import static org.apache.arrow.flight.sql.FlightSQLUtils.FLIGHT_SQL_ACTIONS;
+import static org.apache.arrow.flight.sql.FlightSQLUtils.FLIGHT_SQL_CLOSEPREPAREDSTATEMENT;
+import static org.apache.arrow.flight.sql.FlightSQLUtils.FLIGHT_SQL_GETCATALOGS;
+import static org.apache.arrow.flight.sql.FlightSQLUtils.FLIGHT_SQL_GETPREPAREDSTATEMENT;
+import static org.apache.arrow.flight.sql.FlightSQLUtils.FLIGHT_SQL_GETSCHEMAS;
+import static org.apache.arrow.flight.sql.FlightSQLUtils.FLIGHT_SQL_GETSQLINFO;
+import static org.apache.arrow.flight.sql.FlightSQLUtils.FLIGHT_SQL_GETTABLES;
+import static org.apache.arrow.flight.sql.FlightSQLUtils.FLIGHT_SQL_GETTABLETYPES;
+
+import org.apache.arrow.flight.Action;
+import org.apache.arrow.flight.ActionType;
+import org.apache.arrow.flight.FlightDescriptor;
+import org.apache.arrow.flight.FlightInfo;
+import org.apache.arrow.flight.FlightProducer;
+import org.apache.arrow.flight.FlightStream;
+import org.apache.arrow.flight.PutResult;
+import org.apache.arrow.flight.Result;
+import org.apache.arrow.flight.SchemaResult;
+import org.apache.arrow.flight.Ticket;
+import org.apache.arrow.flight.sql.impl.FlightSQL.ActionClosePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSQL.ActionGetCatalogsRequest;
+import org.apache.arrow.flight.sql.impl.FlightSQL.ActionGetPreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSQL.ActionGetSchemasRequest;
+import org.apache.arrow.flight.sql.impl.FlightSQL.ActionGetTablesRequest;
+import org.apache.arrow.flight.sql.impl.FlightSQL.CommandPreparedStatementQuery;
+import org.apache.arrow.flight.sql.impl.FlightSQL.CommandPreparedStatementUpdate;
+import org.apache.arrow.flight.sql.impl.FlightSQL.CommandStatementQuery;
+import org.apache.arrow.flight.sql.impl.FlightSQL.CommandStatementUpdate;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+
+import io.grpc.Status;
+
+/**
+ * API to Implement an Arrow Flight SQL producer.
+ */
+public abstract class FlightSQLProducer implements FlightProducer, AutoCloseable {
+ /**
+ * Depending on the provided command, method either:
+ * 1. Return information about a SQL query, or
+ * 2. Return information about a prepared statement. In this case, parameters binding is allowed.
+ *
+ * @param context Per-call context.
+ * @param descriptor The descriptor identifying the data stream.
+ * @return information about the given SQL query, or the given prepared statement.
+ */
+ @Override
+ public FlightInfo getFlightInfo(CallContext context, FlightDescriptor descriptor) {
+ final Any command = FlightSQLUtils.parseOrThrow(descriptor.getCommand());
Review comment:
Instead of Any.pack above, what if we define an explicit top-level message that has the possible messages in a oneof?
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
Review comment:
Is this an AND or an OR? Are these filters matched by substring or with the pattern syntax above?
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 6;
+}
+
+/*
+ * Wrap the result of a "GetTables" action.
+ */
+message ActionGetTablesResult {
+ string catalog = 1;
+ string schema = 2;
+ string table = 3;
+ string table_type = 4;
+
+ /*
+ * Schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ * Null if include_schema on request is false.
+ */
+ bytes arrow_metadata = 5;
+}
+
+/*
+ * Wrap the result of a "GetTableTypes" action.
+ */
+message ActionGetTableTypesResult {
+ /*
+ * Indicates the type of the table. E.g. table (regular data table) , view, system table etc.
+ */
+ string table_type = 1;
+}
+
+// SQL Execution Action Messages
+
+/*
+ * Request message for the "GetPreparedStatement" action on a
+ * Flight SQL enabled backend.
+ */
+message ActionGetPreparedStatementRequest {
+ // The valid SQL string to get a prepared statement for.
+ string query = 1;
+}
+
+/*
+ * Wrap the result of a "GetPreparedStatement" action.
+ */
+message ActionGetPreparedStatementResult {
+
+ // Opaque handle for the prepared statement on the server.
+ bytes prepared_statement_handle = 1;
+
+ // If a result set generating query was provided, dataset_schema contains the
+ // schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ bytes dataset_schema = 2;
+
+ // If the query provided contained parameters, parameter_schema contains the
+ // Schema of the expected parameters as described in Schema.fbs::Schema.
+ bytes parameter_schema = 3;
+}
+
+/*
+ * Request message for the "ClosePreparedStatement" action on a
+ * Flight SQL enabled backend.
+ * Closes server resources associated with the prepared statement handle.
+ */
+message ActionClosePreparedStatementRequest {
+ // Opaque handle for the prepared statement on the server.
+ string prepared_statement_handle = 1;
+}
+
+
+// SQL Execution Messages.
+
+/*
+ * Represents a SQL query. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ */
+message CommandStatementQuery {
+ // The SQL syntax.
+ string query = 1;
+}
+
+/*
+ * Represents an instance of executing a prepared statement. Used in the
+ * command member of FlightDescriptor for the following RPC calls:
+ * TODO: Is the idea that a Put with parameter values would execute multiple bound versions of the prepared statement
+ * TODO: (one for each row)? Seems like that will work ok for Insert statements, but what about other kinds of prepared statements?
+ * - DoPut: bind parameter values.
+ * - GetFlightInfo: execute the prepared statement instance.
+ */
+message CommandPreparedStatementQuery {
+ // Unique identifier for the instance of the prepared statement to execute.
+ bytes client_execution_handle = 1;
+ // Opaque handle for the prepared statement on the server.
+ bytes prepared_statement_handle = 2;
+}
+
+/*
+ * Represents a SQL update query. Used in the command member of FlightDescriptor
+ * for the the RPC call DoPut to cause the server to execute the included
+ * SQL update.
+ */
+message CommandStatementUpdate {
+ // The SQL syntax.
+ string query = 1;
+}
+
+/*
+ * Represents a SQL update query. Used in the command member of FlightDescriptor
+ * for the the RPC call DoPut to cause the server to execute the included
+ * prepared statement handle as an update.
+ */
+message CommandPreparedStatementUpdate {
+ // Unique identifier for the instance of the prepared statement to execute.
+ bytes client_execution_handle = 1;
+ // Opaque handle for the prepared statement on the server.
+ bytes prepared_statement_handle = 2;
+}
+
+/*
+ * Returned from the RPC call DoPut when a CommandStatementUpdate
+ * CommandPreparedStatementUpdate was in the request, containing
+ * results from the update.
+ */
+message DoPutUpdateResult {
+ // The number of records updated. A return value of -1 represents
+ // an unknown updated record count.
+ int64 record_count = 1;
+}
+
+message ResultsOrder {
Review comment:
Shouldn't this message have a field?
Alternatively, if you wanted a top-level enum, you could just have
```
enum ResultsOrder {
RESULTS_ORDER_UNKNOWN = 0;
// ...
}
```
I would also suggest prefixing enum variants with the enum name since Protobuf enums aren't namespaced.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r652306569
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
Review comment:
would using information_schema combine the two?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r660991415
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
+
+ // Implementation specific custom SqlInfo values.
+ repeated string custom_info = 2;
+}
+
+/*
+ Represents a request to retrieve the list of catalogs on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name.
+ */
+message CommandGetCatalogs {
+}
+
+/*
+ * Represents a request to retrieve the list of schemas on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name, then schema_name.
+ */
+message CommandGetSchemas {
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then all catalogs are searched.
+ */
+ string catalog = 1;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 2;
+}
+
+/*
+ * Represents a request to retrieve the list of tables, and optionally their schemas, on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8,
+ * table_name: utf8,
+ * table_type: utf8,
+ * table_schema: bytes
+ * >
+ * The returned data should be ordered by catalog_name, schema_name, table_name, then table_type.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then all catalogs are searched.
+ */
+ string catalog = 1;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, all schemas matching other filters are searched.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 2;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no table_name_filter_pattern is provided, all tables matching other filters are searched.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 3;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 4;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 5;
+}
+
+/*
+ * Represents a request to retrieve the list of table types on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * table_type: utf8
+ * >
+ * The returned data should be ordered by table_type.
+ */
+message CommandGetTableTypes {
+}
+
+// SQL Execution Action Messages
+
+/*
+ * Request message for the "GetPreparedStatement" action on a Flight SQL enabled backend.
+ */
+message ActionCreatePreparedStatementRequest {
+ // The valid SQL string to create a prepared statement for.
+ string query = 1;
+}
+
+/*
+ * Wrap the result of a "GetPreparedStatement" action.
+ */
+message ActionCreatePreparedStatementResult {
+ // Opaque handle for the prepared statement on the server.
+ bytes prepared_statement_handle = 1;
+
+ // If a result set generating query was provided, dataset_schema contains the
+ // schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ bytes dataset_schema = 2;
+
+ // If the query provided contained parameters, parameter_schema contains the
+ // Schema of the expected parameters as described in Schema.fbs::Schema.
+ bytes parameter_schema = 3;
+}
+
+/*
+ * Request message for the "ClosePreparedStatement" action on a Flight SQL enabled backend.
+ * Closes server resources associated with the prepared statement handle.
+ */
+message ActionClosePreparedStatementRequest {
+ // Opaque handle for the prepared statement on the server.
+ string prepared_statement_handle = 1;
+}
+
+
+// SQL Execution Messages.
+
+/*
+ * Represents a SQL query. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ */
+message CommandStatementQuery {
+ // The SQL syntax.
+ string query = 1;
+}
+
+/*
+ * Represents an instance of executing a prepared statement. Used in the command member of FlightDescriptor for
+ * the following RPC calls:
+ * - DoPut: bind parameter values.
+ * - GetFlightInfo: execute the prepared statement instance.
+ */
Review comment:
Yes, but probably not captured:
- All bound parameters are executed as an atomic unit on execute, even if there are multiple parameter sets.
- One row of data per parameter set should work fine for both single parameter sets and multiple parameter sets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r634803104
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
Review comment:
table_types should be OR'd together (same for ODBC/JDBC). Pseudo-tables that require full SQL support may be too difficult for vendors to adopt, but if we permit implementations to support just the filters that are proposed here it reduces the burden a fair bit.
I think just submitting metadata requests via SQL has other challenges from an implementation perspective. It necessitates either having the backend validate parsing of the pseudo table (essentially making it a real table/view) or having the FlightProducer parse the query, identify the pseudo table and its filters, then translating the request to a backend call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664913070
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSqlUtils.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.sql.Types;
+import java.util.List;
+
+import org.apache.arrow.flight.ActionType;
+import org.apache.arrow.vector.types.DateUnit;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+
+import com.google.common.collect.ImmutableList;
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+import com.google.protobuf.Message;
+
+/**
+ * Utilities to work with Flight SQL semantics.
+ */
+public final class FlightSqlUtils {
+
+ private static final int BIT_WIDTH8 = 8;
+ private static final int BIT_WIDTH_16 = 16;
+ private static final int BIT_WIDTH_32 = 32;
+ private static final int BIT_WIDTH_64 = 64;
+ private static final boolean IS_SIGNED_FALSE = false;
+ private static final boolean IS_SIGNED_TRUE = true;
+
+ public static final ActionType FLIGHT_SQL_CREATEPREPAREDSTATEMENT = new ActionType("CreatePreparedStatement",
+ "Creates a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionCreatePreparedStatementRequest\n" +
+ "Response Message: ActionCreatePreparedStatementResult");
+
+ public static final ActionType FLIGHT_SQL_CLOSEPREPAREDSTATEMENT = new ActionType("ClosePreparedStatement",
+ "Closes a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionClosePreparedStatementRequest\n" +
+ "Response Message: N/A");
+
+ public static final List<ActionType> FLIGHT_SQL_ACTIONS = ImmutableList.of(
+ FLIGHT_SQL_CREATEPREPAREDSTATEMENT,
+ FLIGHT_SQL_CLOSEPREPAREDSTATEMENT
+ );
+
+ /**
+ * Converts {@link java.sql.Types} values returned from JDBC Apis to Arrow types.
+ *
+ * @param jdbcDataType {@link java.sql.Types} value.
+ * @param precision Precision of the type.
+ * @param scale Scale of the type.
+ * @return The Arrow equivalent type.
+ */
+ public static ArrowType getArrowTypeFromJDBCType(int jdbcDataType, int precision, int scale) {
Review comment:
This method leaks JDBC into Arrow. I believe an earlier comment mentioned there's a JdbcToArrow converter component that had a similar method that we can use instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r635404836
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSQLClientUtils.java
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.arrow.flight.Action;
+import org.apache.arrow.flight.FlightClient;
+import org.apache.arrow.flight.FlightDescriptor;
+import org.apache.arrow.flight.FlightInfo;
+import org.apache.arrow.flight.Result;
+import org.apache.arrow.flight.sql.impl.FlightSQL;
+import org.apache.arrow.flight.sql.impl.FlightSQL.ActionGetPreparedStatementResult;
+import org.apache.arrow.flight.sql.impl.FlightSQL.ActionGetTablesRequest;
+import org.apache.arrow.flight.sql.impl.FlightSQL.ActionGetTablesResult;
+import org.apache.arrow.flight.sql.impl.FlightSQL.CommandPreparedStatementQuery;
+import org.apache.arrow.vector.types.pojo.Schema;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.ByteString;
+
+import io.grpc.Status;
+
+/**
+ * Client side utilities to work with Flight SQL semantics.
+ */
+public final class FlightSQLClientUtils {
+
+ /**
+ * Helper method to request a list of tables from a Flight SQL enabled endpoint.
+ *
+ * @param client The Flight Client.
+ * @param catalog The catalog.
+ * @param schemaFilterPattern The schema filter pattern.
+ * @param tableFilterPattern The table filter pattern.
+ * @param tableTypes The table types to include.
+ * @param includeSchema True to include the schema upon return, false to not include the schema.
+ * @return A list of tables matching the criteria.
+ */
+ public static List<ActionGetTablesResult> getTables(FlightClient client, String catalog, String schemaFilterPattern,
+ String tableFilterPattern, List<String> tableTypes, boolean includeSchema) {
+
+ final ActionGetTablesRequest.Builder requestBuilder = ActionGetTablesRequest
+ .newBuilder()
+ .setIncludeSchema(includeSchema);
+
+ if (catalog != null) {
+ requestBuilder.setCatalog(catalog);
+ }
+
+ if (schemaFilterPattern != null) {
+ requestBuilder.setSchemaFilterPattern(schemaFilterPattern);
+ }
+
+ if (tableFilterPattern != null) {
+ requestBuilder.setTableNameFilterPattern(tableFilterPattern);
+ }
+
+ if (tableTypes != null) {
+ requestBuilder.addAllTableTypes(tableTypes);
+ }
+
+ final Iterator<Result> results = client.doAction(new Action(
+ "GetTables", Any.pack(requestBuilder.build()).toByteArray()));
+
+ final List<ActionGetTablesResult> getTablesResults = new ArrayList<>();
+ results.forEachRemaining(result -> {
Review comment:
This should be done lazily instead of caching everything in an ArrayList.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664935069
##########
File path: java/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/FlightSqlExample.java
##########
@@ -0,0 +1,599 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import static org.apache.arrow.flight.sql.FlightSqlUtils.getArrowTypeFromJDBCType;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ParameterMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.ResultSetMetaData;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.ExecutionException;
+import java.util.stream.Stream;
+
+import org.apache.arrow.flight.CallStatus;
+import org.apache.arrow.flight.Criteria;
+import org.apache.arrow.flight.FlightDescriptor;
+import org.apache.arrow.flight.FlightEndpoint;
+import org.apache.arrow.flight.FlightInfo;
+import org.apache.arrow.flight.FlightRuntimeException;
+import org.apache.arrow.flight.FlightStatusCode;
+import org.apache.arrow.flight.FlightStream;
+import org.apache.arrow.flight.Location;
+import org.apache.arrow.flight.PutResult;
+import org.apache.arrow.flight.Result;
+import org.apache.arrow.flight.SchemaResult;
+import org.apache.arrow.flight.Ticket;
+import org.apache.arrow.flight.sql.impl.FlightSql;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementQuery;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementUpdate;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandStatementQuery;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandStatementUpdate;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.util.AutoCloseables;
+import org.apache.arrow.util.Preconditions;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.dictionary.DictionaryProvider;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.commons.dbcp2.ConnectionFactory;
+import org.apache.commons.dbcp2.DriverManagerConnectionFactory;
+import org.apache.commons.dbcp2.PoolableConnection;
+import org.apache.commons.dbcp2.PoolableConnectionFactory;
+import org.apache.commons.dbcp2.PoolingDataSource;
+import org.apache.commons.pool2.ObjectPool;
+import org.apache.commons.pool2.impl.GenericObjectPool;
+
+import com.google.common.cache.CacheBuilder;
+import com.google.common.cache.CacheLoader;
+import com.google.common.cache.LoadingCache;
+import com.google.common.cache.RemovalListener;
+import com.google.common.cache.RemovalNotification;
+import com.google.common.collect.ImmutableList;
+import com.google.protobuf.Any;
+import com.google.protobuf.ByteString;
+import com.google.protobuf.InvalidProtocolBufferException;
+
+import io.grpc.Status;
+
+/**
+ * Proof of concept {@link FlightSqlProducer} implementation showing an Apache Derby backed Flight SQL server capable
+ * of the following workflows:
+ * - returning a list of tables from the action "GetTables".
+ * - creation of a prepared statement from the action "GetPreparedStatement".
+ * - execution of a prepared statement by using a {@link CommandPreparedStatementQuery} with getFlightInfo and
+ * getStream.
+ */
+public class FlightSqlExample extends FlightSqlProducer implements AutoCloseable {
+ private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(FlightSqlExample.class);
+
+ private static final int BATCH_ROW_SIZE = 1000;
+
+ private final Location location;
+ private final PoolingDataSource<PoolableConnection> dataSource;
+
+ private final LoadingCache<CommandPreparedStatementQuery, ResultSet> commandExecutePreparedStatementLoadingCache;
+ private final LoadingCache<PreparedStatementCacheKey, PreparedStatementContext> preparedStatementLoadingCache;
+
+ public FlightSqlExample(Location location) {
+ removeDerbyDatabaseIfExists();
+ populateDerbyDatabase();
+
+ final ConnectionFactory connectionFactory =
+ new DriverManagerConnectionFactory("jdbc:derby:target/derbyDB", null);
+ final PoolableConnectionFactory poolableConnectionFactory = new PoolableConnectionFactory(connectionFactory, null);
+ final ObjectPool<PoolableConnection> connectionPool = new GenericObjectPool<>(poolableConnectionFactory);
+ poolableConnectionFactory.setPool(connectionPool);
+
+ // PoolingDataSource takes ownership of connectionPool.
+ dataSource = new PoolingDataSource<>(connectionPool);
+
+ preparedStatementLoadingCache =
+ CacheBuilder.newBuilder()
+ .maximumSize(100)
+ .expireAfterWrite(10, java.util.concurrent.TimeUnit.MINUTES)
+ .removalListener(new PreparedStatementRemovalListener())
+ .build(new PreparedStatementCacheLoader(dataSource));
+
+ commandExecutePreparedStatementLoadingCache =
+ CacheBuilder.newBuilder()
+ .maximumSize(100)
+ .expireAfterWrite(10, java.util.concurrent.TimeUnit.MINUTES)
+ .removalListener(new CommandExecutePreparedStatementRemovalListener())
+ .build(new CommandExecutePreparedStatementCacheLoader(preparedStatementLoadingCache));
+
+ this.location = location;
+ }
+
+ @Override
+ public void getTables(FlightSql.ActionGetTablesRequest request, CallContext context,
+ StreamListener<Result> listener) {
+ try {
+ final String catalog = (request.getCatalog().isEmpty() ? null : request.getCatalog());
+
+ final String schemaFilterPattern =
+ (request.getSchemaFilterPattern().isEmpty() ? null : request.getSchemaFilterPattern());
+
+ final String tableFilterPattern =
+ (request.getTableNameFilterPattern().isEmpty() ? null : request.getTableNameFilterPattern());
+
+ final String[] tableTypes = request.getTableTypesList().size() == 0 ? null :
+ request.getTableTypesList().toArray(new String[request.getTableTypesList().size()]);
+
+ try (final Connection connection = dataSource.getConnection();
+ final ResultSet tables = connection.getMetaData().getTables(
+ catalog,
+ schemaFilterPattern,
+ tableFilterPattern,
+ tableTypes)) {
+ while (tables.next()) {
+ listener.onNext(getTableResult(tables, request.getIncludeSchema()));
+ }
+ }
+ } catch (SQLException e) {
+ listener.onError(e);
+ } finally {
+ listener.onCompleted();
+ }
+ }
+
+ private Result getTableResult(final ResultSet tables, boolean includeSchema) throws SQLException {
+
+ final String catalog = tables.getString("TABLE_CAT");
+ final String schema = tables.getString("TABLE_SCHEM");
+ final String table = tables.getString("TABLE_NAME");
+ final String tableType = tables.getString("TABLE_TYPE");
+
+ final ActionGetTablesResult.Builder builder = ActionGetTablesResult.newBuilder()
+ .setCatalog(catalog)
+ .setSchema(schema)
+ .setTable(table)
+ .setTableType(tableType);
+
+ if (includeSchema) {
+ final Schema pojoSchema = buildSchema(catalog, schema, table);
+ builder.setArrowMetadata(ByteString.copyFrom(pojoSchema.toByteArray()));
+ }
+
+ return new Result(Any.pack(builder.build()).toByteArray());
+ }
+
+ @Override
+ public void getPreparedStatement(FlightSql.ActionGetPreparedStatementRequest request, CallContext context,
+ StreamListener<Result> listener) {
+ final PreparedStatementCacheKey handle = new PreparedStatementCacheKey(
+ UUID.randomUUID().toString(), request.getQuery());
+
+ try {
+ final PreparedStatementContext preparedStatementContext = preparedStatementLoadingCache.get(handle);
+ final PreparedStatement preparedStatement = preparedStatementContext.getPreparedStatement();
+
+ // todo
+ final Schema pojoParameterMetaDataSchema = buildSchema(preparedStatement.getParameterMetaData());
+ final Schema pojoResultSetSchema = buildSchema(preparedStatement.getMetaData());
+
+ listener.onNext(new Result(
+ Any.pack(ActionGetPreparedStatementResult.newBuilder()
+ .setDatasetSchema(ByteString.copyFrom(pojoResultSetSchema.toByteArray()))
+ .setParameterSchema(ByteString.copyFrom(pojoParameterMetaDataSchema.toByteArray()))
+ .setPreparedStatementHandle(handle.toProtocol())
+ .build())
+ .toByteArray()));
+
+ } catch (ExecutionException | SQLException e) {
+ listener.onError(e);
+ } finally {
+ listener.onCompleted();
+ }
+ }
+
+ @Override
+ public FlightInfo getFlightInfoPreparedStatement(CommandPreparedStatementQuery command, FlightDescriptor descriptor,
+ CallContext context) {
+ try {
+ final ResultSet resultSet = commandExecutePreparedStatementLoadingCache.get(command);
+ final Schema schema = buildSchema(resultSet.getMetaData());
+
+ final List<FlightEndpoint> endpoints = ImmutableList
+ .of(new FlightEndpoint(new Ticket(Any.pack(command).toByteArray()), location));
+
+ return new FlightInfo(schema, descriptor, endpoints, -1, -1);
+ } catch (ExecutionException | SQLException e) {
+ logger.error("There was a problem executing the prepared statement", e);
+ throw new FlightRuntimeException(new CallStatus(FlightStatusCode.INTERNAL, e, e.getMessage(), null));
+ }
+ }
+
+ private Schema buildSchema(String catalog, String schema, String table) throws SQLException {
+ final List<Field> fields = new ArrayList<>();
+
+ try (final Connection connection = dataSource.getConnection();
+ final ResultSet columns = connection.getMetaData().getColumns(
+ catalog,
+ schema,
+ table,
+ null);) {
+
+ while (columns.next()) {
+ final String columnName = columns.getString("COLUMN_NAME");
+ final int jdbcDataType = columns.getInt("DATA_TYPE");
+ final String jdbcDataTypeName = columns.getString("TYPE_NAME");
+ final String jdbcIsNullable = columns.getString("IS_NULLABLE");
+ final boolean arrowIsNullable = jdbcIsNullable.equals("YES");
+
+ final int precision = columns.getInt("DECIMAL_DIGITS");
+ final int scale = columns.getInt("COLUMN_SIZE");
+ final ArrowType arrowType = FlightSqlUtils.getArrowTypeFromJDBCType(jdbcDataType, precision, scale);
+
+ final FieldType fieldType = new FieldType(arrowIsNullable, arrowType, /*dictionary=*/null);
+ fields.add(new Field(columnName, fieldType, null));
+ }
+ }
+
+ return new Schema(fields);
+ }
+
+ @Override
+ public void getStreamPreparedStatement(CommandPreparedStatementQuery command, CallContext context, Ticket ticket,
+ ServerStreamListener listener) {
+ try {
+ final ResultSet resultSet = commandExecutePreparedStatementLoadingCache.get(command);
+ final ResultSetMetaData resultSetMetaData = resultSet.getMetaData();
+ final Schema schema = buildSchema(resultSetMetaData);
+ final DictionaryProvider dictionaryProvider = new DictionaryProvider.MapDictionaryProvider();
+
+ try (final BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+ final VectorSchemaRoot root = VectorSchemaRoot.create(schema, allocator)) {
+
+ listener.start(root, dictionaryProvider);
+ final int columnCount = resultSetMetaData.getColumnCount();
+
+ while (resultSet.next()) {
+ final int rowCounter = readBatch(resultSet, resultSetMetaData, root, columnCount);
+
+ for (int resultSetColumnCounter = 1; resultSetColumnCounter <= columnCount; resultSetColumnCounter++) {
+ final String columnName = resultSetMetaData.getColumnName(resultSetColumnCounter);
+ root.getVector(columnName).setValueCount(rowCounter);
+ }
+
+ root.setRowCount(rowCounter);
+ listener.putNext();
+ }
+ }
+ } catch (ExecutionException | SQLException e) {
Review comment:
I changed it to be Throwable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r567492292
##########
File path: java/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/PreparedStatementContext.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.util.Objects;
+
+import org.apache.arrow.util.AutoCloseables;
+
+class PreparedStatementContext implements AutoCloseable {
Review comment:
I think instead of placing this in tests, we should have a separate top level package for a JDBC reference implementation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r573043365
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
Review comment:
SQL first approach:
` __APACHE_ARROW_FLIGHT_SCHEMAS`:
```
catalog: utf8
schema: utf8
```
Minimal SQL supported. Projection, Ordering (might be limited to required catalog and schema) and LIKE or equality clauses on schema field.
i.e.
```
select catalog, schema from __APACHE_ARROW_FLIGHT_SCHEMAS
order by catalog, schema descending
where schema LIKE 'A%'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-871553639
@kylepbit yes, agreed. Thanks for the updates!
I'll try and remember to approve/kick off the CI when updated but also feel free to ping me to do so.
I'll note that Micah left one comment way at the top about naming the file `FlightSql.proto` instead of `FlightSQL.proto` for consistency (I think doing that rename might lose the comments so now is probably an OK time to do it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664934784
##########
File path: java/flight/flight-sql/src/test/java/org/apache/arrow/flight/TestFlightSql.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight;
+
+import static org.apache.arrow.flight.sql.FlightSqlClientUtils.getPreparedStatement;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.arrow.flight.sql.FlightSqlClientUtils;
+import org.apache.arrow.flight.sql.FlightSqlExample;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionClosePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionCreatePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionCreatePreparedStatementResult;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementQuery;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.memory.util.ArrowBufPointer;
+import org.apache.arrow.util.AutoCloseables;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.arrow.vector.util.ElementAddressableVectorIterator;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.ByteString;
+
+/**
+ * Test direct usage of Flight SQL workflows.
+ */
+public class TestFlightSql {
+ private static BufferAllocator allocator;
+ private static FlightServer server;
+
+ private static FlightClient client;
+
+ protected static final Schema SCHEMA_INT_TABLE = new Schema(Arrays.asList(
+ new Field("KEYNAME", new
+ FieldType(true, ArrowType.Utf8.INSTANCE, null),
+ null),
+ new Field("VALUE",
+ new FieldType(true, new ArrowType.Int(32, true), null),
+ null)));
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+ allocator = new RootAllocator(Integer.MAX_VALUE);
+
+ final Location serverLocation = Location.forGrpcInsecure(FlightTestUtil.LOCALHOST, 0);
+ server = FlightServer.builder(allocator, serverLocation, new FlightSqlExample(serverLocation)).build();
+ server.start();
+
+ final Location clientLocation = Location.forGrpcInsecure(FlightTestUtil.LOCALHOST, server.getPort());
+ client = FlightClient.builder(allocator, clientLocation).build();
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ AutoCloseables.close(client, server, allocator);
+ }
+
+ /*
+ @Test
+ public void testGetTables() throws Exception {
+ // Arrange
+ final ActionGetTablesResult expected = ActionGetTablesResult.newBuilder()
+ .setSchema("APP")
+ .setTable("INTTABLE")
+ .setTableType("TABLE")
+ .setArrowMetadata(ByteString.copyFrom(SCHEMA_INT_TABLE.toByteArray()))
+ .build();
+
+ // Act
+ final Iterator<Result> results = client.doAction(new Action("GetTables",
+ Any.pack(ActionGetTablesRequest
+ .newBuilder()
+ .addTableTypes("TABLE")
+ .setIncludeSchema(true)
+ .build())
+ .toByteArray()));
+
+ // Assert
+ while (results.hasNext()) {
+ ActionGetTablesResult actual = Any.parseFrom(results.next().getBody()).unpack(ActionGetTablesResult.class);
+ assertEquals(expected, actual);
+ }
+ }
+
+ @Test
+ public void testGetTablesWithFlightSqlClientUtils() throws Exception {
+ // Arrange
+ final ActionGetTablesResult expected = ActionGetTablesResult.newBuilder()
+ .setSchema("APP")
+ .setTable("INTTABLE")
+ .setTableType("TABLE")
+ .setArrowMetadata(ByteString.copyFrom(SCHEMA_INT_TABLE.toByteArray()))
+ .build();
+
+ // Act
+ final List<ActionGetTablesResult> results = getTables(client, null, null, null,
+ Collections.singletonList("TABLE"), true);
+
+ // Assert
+ assertEquals(1, results.size());
+ assertEquals(expected, results.get(0));
+ }
+ */
+
+ @Test
+ public void testSimplePrepStmt() throws Exception {
+ final Iterator<Result> preparedStatementResults = client.doAction(new Action("GetPreparedStatement",
+ Any.pack(ActionCreatePreparedStatementRequest
+ .newBuilder()
+ .setQuery("Select * from intTable")
+ .build())
+ .toByteArray()));
+
+ assertTrue(preparedStatementResults.hasNext());
+ final ActionCreatePreparedStatementResult preparedStatementResult =
+ Any.parseFrom(preparedStatementResults.next().getBody()).unpack(ActionCreatePreparedStatementResult.class);
+ assertFalse(preparedStatementResults.hasNext());
+
+ final Schema actualSchema = Schema.deserialize(preparedStatementResult.getDatasetSchema().asReadOnlyByteBuffer());
+ assertEquals(SCHEMA_INT_TABLE, actualSchema);
+
+ final FlightDescriptor descriptor = FlightDescriptor
+ .command(Any.pack(CommandPreparedStatementQuery.newBuilder()
+ .setClientExecutionHandle(ByteString.copyFrom(new byte[]{1, 2, 3, 4}))
+ .setPreparedStatementHandle(preparedStatementResult.getPreparedStatementHandle())
+ .build())
+ .toByteArray());
+
+ final FlightInfo info = client.getInfo(descriptor);
+ assertEquals(SCHEMA_INT_TABLE, info.getSchema());
+
+ final FlightStream stream = client.getStream(info.getEndpoints().get(0).getTicket());
+ assertEquals(SCHEMA_INT_TABLE, stream.getSchema());
+
+ List<String> actualStringResults = new ArrayList<>();
+ List<Integer> actualIntResults = new ArrayList<>();
+ while (stream.next()) {
+ final VectorSchemaRoot root = stream.getRoot();
+ final long rowCount = root.getRowCount();
+
+ for (Field field : root.getSchema().getFields()) {
+ final FieldVector fieldVector = root.getVector(field.getName());
+
+ if (fieldVector instanceof VarCharVector) {
+
+ final ElementAddressableVectorIterator<VarCharVector> it =
+ new ElementAddressableVectorIterator<>((VarCharVector) fieldVector);
+
+ for (int rowIndex = 0; rowIndex < rowCount; rowIndex++) {
+ final ArrowBufPointer pt = it.next();
+ final byte[] bytes = new byte[(int) pt.getLength()];
+ pt.getBuf().getBytes(pt.getOffset(), bytes);
+
+ actualStringResults.add(new String(bytes, StandardCharsets.UTF_8));
+ }
+ } else if (fieldVector instanceof IntVector) {
+ for (int rowIndex = 0; rowIndex < rowCount; rowIndex++) {
+ actualIntResults.add(((IntVector) fieldVector).get(rowIndex));
+ }
+ }
+ }
+ }
+ stream.getRoot().clear();
+
+ assertEquals(Arrays.asList("one", "zero", "negative one"), actualStringResults);
+ assertEquals(Arrays.asList(1, 0, -1), actualIntResults);
+
+ final Iterator<Result> closePreparedStatementResults = client.doAction(new Action("ClosePreparedStatement",
+ Any.pack(ActionClosePreparedStatementRequest
+ .newBuilder()
+ .setPreparedStatementHandleBytes(preparedStatementResult.getPreparedStatementHandle())
+ .build())
+ .toByteArray()));
+ assertFalse(closePreparedStatementResults.hasNext());
+ }
+
+ @Test
+ public void testSimplePrepStmtWithFlightSqlClientUtils() throws Exception {
+ final FlightSqlClientUtils.FlightSqlPreparedStatement preparedStatement =
+ getPreparedStatement(client, "Select * from intTable");
+
+ final Schema actualSchema = preparedStatement.getResultSetSchema();
+ assertEquals(SCHEMA_INT_TABLE, actualSchema);
+
+ final FlightInfo info = preparedStatement.executeQuery();
+ assertEquals(SCHEMA_INT_TABLE, info.getSchema());
+
+ final FlightStream stream = client.getStream(info.getEndpoints().get(0).getTicket());
+ assertEquals(SCHEMA_INT_TABLE, stream.getSchema());
+
+ List<String> actualStringResults = new ArrayList<>();
+ List<Integer> actualIntResults = new ArrayList<>();
+ while (stream.next()) {
+ final VectorSchemaRoot root = stream.getRoot();
+ final long rowCount = root.getRowCount();
+
+ for (Field field : root.getSchema().getFields()) {
+ final FieldVector fieldVector = root.getVector(field.getName());
+
+ if (fieldVector instanceof VarCharVector) {
+
+ final ElementAddressableVectorIterator<VarCharVector> it =
+ new ElementAddressableVectorIterator<>((VarCharVector) fieldVector);
+
+ for (int rowIndex = 0; rowIndex < rowCount; rowIndex++) {
+ final ArrowBufPointer pt = it.next();
+ final byte[] bytes = new byte[(int) pt.getLength()];
+ pt.getBuf().getBytes(pt.getOffset(), bytes);
+
+ actualStringResults.add(new String(bytes, StandardCharsets.UTF_8));
+ }
+ } else if (fieldVector instanceof IntVector) {
+ for (int rowIndex = 0; rowIndex < rowCount; rowIndex++) {
+ actualIntResults.add(((IntVector) fieldVector).get(rowIndex));
+ }
+ }
+ }
+ }
+ stream.getRoot().clear();
+
+ assertEquals(Arrays.asList("one", "zero", "negative one"), actualStringResults);
+ assertEquals(Arrays.asList(1, 0, -1), actualIntResults);
+
+ AutoCloseables.close(preparedStatement);
Review comment:
That was done intentionally to allow for testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r634786239
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
Review comment:
A long is much more suitable for a bit mask given there are 8 bits available then.
I suggest we revisit what types can be used after we've identified a 'standardized' set of GetSqlInfo properties.
I also think it'd be better to use integral type codes instead of string values for keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664909672
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSqlClientUtils.java
##########
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.arrow.flight.Action;
+import org.apache.arrow.flight.FlightClient;
+import org.apache.arrow.flight.FlightDescriptor;
+import org.apache.arrow.flight.FlightInfo;
+import org.apache.arrow.flight.Result;
+import org.apache.arrow.flight.sql.impl.FlightSql;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionCreatePreparedStatementResult;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementQuery;
+import org.apache.arrow.vector.types.pojo.Schema;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.ByteString;
+
+import io.grpc.Status;
+
+/**
+ * Client side utilities to work with Flight SQL semantics.
+ */
+public final class FlightSqlClientUtils {
+
+ /**
+ * Helper method to request a list of tables from a Flight SQL enabled endpoint.
+ *
+ * @param client The Flight Client.
+ * @param catalog The catalog.
+ * @param schemaFilterPattern The schema filter pattern.
+ * @param tableFilterPattern The table filter pattern.
+ * @param tableTypes The table types to include.
+ * @param includeSchema True to include the schema upon return, false to not include the schema.
+ * @return a FlightInfo object representing the stream(s) to fetch.
+ */
+ public static FlightInfo getTables(FlightClient client, String catalog, String schemaFilterPattern,
+ String tableFilterPattern, List<String> tableTypes, boolean includeSchema) {
+
+ final FlightSql.CommandGetTables.Builder builder = FlightSql.CommandGetTables.newBuilder();
+
+ if (catalog != null) {
+ builder.setCatalog(catalog);
+ }
+
+ if (schemaFilterPattern != null) {
+ builder.setSchemaFilterPattern(schemaFilterPattern);
+ }
+
+ if (tableFilterPattern != null) {
+ builder.setTableNameFilterPattern(tableFilterPattern);
+ }
+
+ if (tableTypes != null) {
+ builder.addAllTableTypes(tableTypes);
+ }
+ builder.setIncludeSchema(includeSchema);
+
+ final FlightDescriptor descriptor = FlightDescriptor.command(Any.pack(builder.build()).toByteArray());
+ return client.getInfo(descriptor);
+ }
+
+ /**
+ * Helper method to create a prepared statement on the server.
+ *
+ * @param client The Flight Client.
+ * @param query The query to prepare.
+ * @return Metadata and handles to the prepared statement which exists on the server.
+ */
+ public static FlightSqlPreparedStatement getPreparedStatement(FlightClient client, String query) {
+ return new FlightSqlPreparedStatement(client, query);
+ }
+
+ /**
+ * Helper class to encapsulate Flight SQL prepared statement logic.
+ */
+ public static class FlightSqlPreparedStatement implements Closeable {
+ private final FlightClient client;
+ private final ActionCreatePreparedStatementResult preparedStatementResult;
+ private long invocationCount;
+ private boolean isClosed;
+ private Schema resultSetSchema = null;
+ private Schema parameterSchema = null;
+
+ /**
+ * Constructor.
+ *
+ * @param client The client. FlightSqlPreparedStatement does not maintain this resource.
+ * @param sql The query.
+ */
+ public FlightSqlPreparedStatement(FlightClient client, String sql) {
+ this.client = client;
+
+ final Iterator<Result> preparedStatementResults = client.doAction(new Action("GetPreparedStatement",
+ Any.pack(FlightSql.ActionCreatePreparedStatementRequest
+ .newBuilder()
+ .setQuery(sql)
+ .build())
+ .toByteArray()));
+
+ preparedStatementResult = FlightSqlUtils.unpackAndParseOrThrow(
+ preparedStatementResults.next().getBody(),
+ ActionCreatePreparedStatementResult.class);
+
+ invocationCount = 0;
+ isClosed = false;
+ }
+
+ /**
+ * Returns the Schema of the resultset.
+ *
+ * @return the Schema of the resultset.
+ */
+ public Schema getResultSetSchema() {
+ if (resultSetSchema == null && preparedStatementResult.getDatasetSchema() != null) {
+ resultSetSchema = Schema.deserialize(preparedStatementResult.getDatasetSchema().asReadOnlyByteBuffer());
+ }
+ return resultSetSchema;
+ }
+
+ /**
+ * Returns the Schema of the parameters.
+ *
+ * @return the Schema of the parameters.
+ */
+ public Schema getParameterSchema() {
+ if (parameterSchema == null && preparedStatementResult.getParameterSchema() != null) {
+ parameterSchema = Schema.deserialize(preparedStatementResult.getParameterSchema().asReadOnlyByteBuffer());
+ }
+ return parameterSchema;
+ }
+
+ /**
+ * Executes the prepared statement query on the server.
+ *
+ * @return a FlightInfo object representing the stream(s) to fetch.
+ * @throws IOException if the PreparedStatement is closed.
+ */
+ public FlightInfo executeQuery() throws IOException {
+ if (isClosed) {
+ throw new IOException("Prepared statement has already been closed on the server.");
Review comment:
Nit: This isn't an IO error and is more of an IllegalStateException.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield edited a comment on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield edited a comment on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-770164877
Small style nit: Generally camel casing even for acronyms is preferred (i.e. FlightSql.proto, there is probably inconsistency within the code base but I think most new code uses this convention)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664912575
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSqlUtils.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.sql.Types;
+import java.util.List;
+
+import org.apache.arrow.flight.ActionType;
+import org.apache.arrow.vector.types.DateUnit;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+
+import com.google.common.collect.ImmutableList;
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+import com.google.protobuf.Message;
+
+/**
+ * Utilities to work with Flight SQL semantics.
+ */
+public final class FlightSqlUtils {
+
+ private static final int BIT_WIDTH8 = 8;
Review comment:
Should be BIT_WIDTH_8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r654545089
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 6;
+}
+
+/*
+ * Wrap the result of a "GetTables" action.
+ */
+message ActionGetTablesResult {
+ string catalog = 1;
+ string schema = 2;
+ string table = 3;
+ string table_type = 4;
+
+ /*
+ * Schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ * Null if include_schema on request is false.
+ */
+ bytes arrow_metadata = 5;
+}
+
+/*
+ * Wrap the result of a "GetTableTypes" action.
+ */
+message ActionGetTableTypesResult {
+ /*
+ * Indicates the type of the table. E.g. table (regular data table) , view, system table etc.
+ */
+ string table_type = 1;
+}
+
+// SQL Execution Action Messages
+
+/*
+ * Request message for the "GetPreparedStatement" action on a
+ * Flight SQL enabled backend.
+ */
+message ActionGetPreparedStatementRequest {
+ // The valid SQL string to get a prepared statement for.
+ string query = 1;
+}
+
+/*
+ * Wrap the result of a "GetPreparedStatement" action.
+ */
+message ActionGetPreparedStatementResult {
+
+ // Opaque handle for the prepared statement on the server.
+ bytes prepared_statement_handle = 1;
+
+ // If a result set generating query was provided, dataset_schema contains the
+ // schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ bytes dataset_schema = 2;
+
+ // If the query provided contained parameters, parameter_schema contains the
+ // Schema of the expected parameters as described in Schema.fbs::Schema.
+ bytes parameter_schema = 3;
+}
+
+/*
+ * Request message for the "ClosePreparedStatement" action on a
+ * Flight SQL enabled backend.
+ * Closes server resources associated with the prepared statement handle.
+ */
+message ActionClosePreparedStatementRequest {
+ // Opaque handle for the prepared statement on the server.
+ string prepared_statement_handle = 1;
+}
+
+
+// SQL Execution Messages.
+
+/*
+ * Represents a SQL query. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ */
+message CommandStatementQuery {
+ // The SQL syntax.
+ string query = 1;
+}
+
+/*
+ * Represents an instance of executing a prepared statement. Used in the
+ * command member of FlightDescriptor for the following RPC calls:
+ * TODO: Is the idea that a Put with parameter values would execute multiple bound versions of the prepared statement
Review comment:
It 'just' needs someone to write it up and propose it. I'll have to check in on the existing thread. I would very much prefer not to require more statefulness if we don't have to, since it'll complicate implementations. (Also, if it's before DoGet, that means the server might have to execute the same query in response to either of two distinct RPC calls?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-843612756
> Thank you for the PR. I think since there isn't a fully implementation example yet for the all the features it would make this easier to review if we cut down the FlightSql proto to what is implemented and focus on each use-case in a separate PR. I think it makes sense to handle read, write and metadata operations separately?
>
> > Like other Flight APIs, flight-sql does not provide implementation details that dictate how a client and server communicates with each other, it simply provides the SQL semantics and apply them onto the Flight API.
>
> IMO, I think we should be be trying to specify more about how client and server interact with each other in terms of error handling and retries, these are important aspects and without it I think some of the utility of standardization will be diminished.
> > Thinking about this a bit more I think an alternative to a lot of the metadata querying is to define a minimal sql table schema and querying capabilities for it. I'll expand more in a little bit
>
> I did a rough sketch on how the metadata layer could be modeled as pseudo-tables as comments on the proto. I think this addresses some uneasiner I had with the existing model:
>
> 1. It uses one consistent data model for tabular data (the arrow model).
> 2. It makes query semantics clearer instead of trying to model them in an ad-hoc manner through protobufs
>
> This comes at slightly more complex for server implementations. And potential confusion on limitations for specific tables. This might be limited to some extend if we had a different GetMetadata action so implementations know to expect a query to specific pseudo-table. I expect we would provide library wrappers over JDBC to also ease implementation.
To clarify, are you suggesting something similar to INFORMATION_SCHEMA?
I am very much in favour of returning the data more like a standard query result. The existing protobuf definitions don't allow vendors to supply extra metadata with their catalog results. (ODBC and JDBC let you add extra columns to getTables() calls).
I am hesitant about using pseudo-tables -- backend databases may not have catalog information in a form that can be transformed by the rest of their SQL engine. It opens up complexity from an implementor's perspective and I'm not sure how beneficial it is from a user perspective.
I'm also OK with making the catalog information accessible with a limited query syntax rather than pseudo-tables (eg SHOW TABLES WHERE ...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664162550
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
Review comment:
Thanks - I think I have it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664914834
##########
File path: java/flight/flight-sql/src/test/java/org/apache/arrow/flight/TestFlightSql.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight;
+
+import static org.apache.arrow.flight.sql.FlightSqlClientUtils.getPreparedStatement;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.arrow.flight.sql.FlightSqlClientUtils;
+import org.apache.arrow.flight.sql.FlightSqlExample;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionClosePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionCreatePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionCreatePreparedStatementResult;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementQuery;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.memory.util.ArrowBufPointer;
+import org.apache.arrow.util.AutoCloseables;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.arrow.vector.util.ElementAddressableVectorIterator;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.ByteString;
+
+/**
+ * Test direct usage of Flight SQL workflows.
+ */
+public class TestFlightSql {
+ private static BufferAllocator allocator;
+ private static FlightServer server;
+
+ private static FlightClient client;
+
+ protected static final Schema SCHEMA_INT_TABLE = new Schema(Arrays.asList(
+ new Field("KEYNAME", new
+ FieldType(true, ArrowType.Utf8.INSTANCE, null),
+ null),
+ new Field("VALUE",
+ new FieldType(true, new ArrowType.Int(32, true), null),
+ null)));
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+ allocator = new RootAllocator(Integer.MAX_VALUE);
+
+ final Location serverLocation = Location.forGrpcInsecure(FlightTestUtil.LOCALHOST, 0);
+ server = FlightServer.builder(allocator, serverLocation, new FlightSqlExample(serverLocation)).build();
+ server.start();
+
+ final Location clientLocation = Location.forGrpcInsecure(FlightTestUtil.LOCALHOST, server.getPort());
+ client = FlightClient.builder(allocator, clientLocation).build();
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ AutoCloseables.close(client, server, allocator);
+ }
+
+ /*
+ @Test
+ public void testGetTables() throws Exception {
+ // Arrange
+ final ActionGetTablesResult expected = ActionGetTablesResult.newBuilder()
+ .setSchema("APP")
+ .setTable("INTTABLE")
+ .setTableType("TABLE")
+ .setArrowMetadata(ByteString.copyFrom(SCHEMA_INT_TABLE.toByteArray()))
+ .build();
+
+ // Act
+ final Iterator<Result> results = client.doAction(new Action("GetTables",
+ Any.pack(ActionGetTablesRequest
+ .newBuilder()
+ .addTableTypes("TABLE")
+ .setIncludeSchema(true)
+ .build())
+ .toByteArray()));
+
+ // Assert
+ while (results.hasNext()) {
+ ActionGetTablesResult actual = Any.parseFrom(results.next().getBody()).unpack(ActionGetTablesResult.class);
+ assertEquals(expected, actual);
+ }
+ }
+
+ @Test
+ public void testGetTablesWithFlightSqlClientUtils() throws Exception {
+ // Arrange
+ final ActionGetTablesResult expected = ActionGetTablesResult.newBuilder()
+ .setSchema("APP")
+ .setTable("INTTABLE")
+ .setTableType("TABLE")
+ .setArrowMetadata(ByteString.copyFrom(SCHEMA_INT_TABLE.toByteArray()))
+ .build();
+
+ // Act
+ final List<ActionGetTablesResult> results = getTables(client, null, null, null,
+ Collections.singletonList("TABLE"), true);
+
+ // Assert
+ assertEquals(1, results.size());
+ assertEquals(expected, results.get(0));
+ }
+ */
+
+ @Test
+ public void testSimplePrepStmt() throws Exception {
+ final Iterator<Result> preparedStatementResults = client.doAction(new Action("GetPreparedStatement",
+ Any.pack(ActionCreatePreparedStatementRequest
+ .newBuilder()
+ .setQuery("Select * from intTable")
+ .build())
+ .toByteArray()));
+
+ assertTrue(preparedStatementResults.hasNext());
+ final ActionCreatePreparedStatementResult preparedStatementResult =
+ Any.parseFrom(preparedStatementResults.next().getBody()).unpack(ActionCreatePreparedStatementResult.class);
+ assertFalse(preparedStatementResults.hasNext());
+
+ final Schema actualSchema = Schema.deserialize(preparedStatementResult.getDatasetSchema().asReadOnlyByteBuffer());
+ assertEquals(SCHEMA_INT_TABLE, actualSchema);
+
+ final FlightDescriptor descriptor = FlightDescriptor
+ .command(Any.pack(CommandPreparedStatementQuery.newBuilder()
+ .setClientExecutionHandle(ByteString.copyFrom(new byte[]{1, 2, 3, 4}))
+ .setPreparedStatementHandle(preparedStatementResult.getPreparedStatementHandle())
+ .build())
+ .toByteArray());
+
+ final FlightInfo info = client.getInfo(descriptor);
+ assertEquals(SCHEMA_INT_TABLE, info.getSchema());
+
+ final FlightStream stream = client.getStream(info.getEndpoints().get(0).getTicket());
Review comment:
try-with-resources
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664136841
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
Review comment:
@lidavidm - how do you actually create a FieldType for a dense union like this? I have the following:
`new ArrowType.Union(UnionMode.Dense, new int[2] {ArrowTypeID.Utf8, ArrowTypeID.Int}), /*dictionary=*/null)`, although I could not find an ArrowTypeID for a bigint, and it seems repetitive to have the int_value and int32_bitmask.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r573058811
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
Review comment:
I do worry it may be a bit of an implementation burden, but maybe that's the kind of thing that the standard adapter defined here can handle. And limiting the available SQL features as you do below helps. Many databases do already expose this kind of metadata in special tables, and the client can just do the necessary conversions from Arrow data to structured data.
I guess - how do we expect these metadata values to be used? Are they in turn going to back a JDBC implementation or a SQLAlchemy connector? That might help inform what we need to expose and how. I'd also be in favor of paring this to the bare minimum and exposing additional metadata as the need becomes clear, instead of trying to map all of JDBC or ODBC in one shot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r573049787
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
Review comment:
see above for another proposal, I think this becomes more elegantly expressed with pseudotables, so we aren't trying to create arbitrary SQL semantics through protobuf. @lidavidm thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r573039463
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
Review comment:
In the SQL approach define a table like:
`__APACHE_FLIGHT_CATALOGS` minimal schema:
```
<catalog_name: utf8>
```
Supports project and optionally sorting on `catalog_name`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664916420
##########
File path: java/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/FlightSqlExample.java
##########
@@ -0,0 +1,599 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import static org.apache.arrow.flight.sql.FlightSqlUtils.getArrowTypeFromJDBCType;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ParameterMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.ResultSetMetaData;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.ExecutionException;
+import java.util.stream.Stream;
+
+import org.apache.arrow.flight.CallStatus;
+import org.apache.arrow.flight.Criteria;
+import org.apache.arrow.flight.FlightDescriptor;
+import org.apache.arrow.flight.FlightEndpoint;
+import org.apache.arrow.flight.FlightInfo;
+import org.apache.arrow.flight.FlightRuntimeException;
+import org.apache.arrow.flight.FlightStatusCode;
+import org.apache.arrow.flight.FlightStream;
+import org.apache.arrow.flight.Location;
+import org.apache.arrow.flight.PutResult;
+import org.apache.arrow.flight.Result;
+import org.apache.arrow.flight.SchemaResult;
+import org.apache.arrow.flight.Ticket;
+import org.apache.arrow.flight.sql.impl.FlightSql;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementQuery;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementUpdate;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandStatementQuery;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandStatementUpdate;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.util.AutoCloseables;
+import org.apache.arrow.util.Preconditions;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.dictionary.DictionaryProvider;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.commons.dbcp2.ConnectionFactory;
+import org.apache.commons.dbcp2.DriverManagerConnectionFactory;
+import org.apache.commons.dbcp2.PoolableConnection;
+import org.apache.commons.dbcp2.PoolableConnectionFactory;
+import org.apache.commons.dbcp2.PoolingDataSource;
+import org.apache.commons.pool2.ObjectPool;
+import org.apache.commons.pool2.impl.GenericObjectPool;
+
+import com.google.common.cache.CacheBuilder;
+import com.google.common.cache.CacheLoader;
+import com.google.common.cache.LoadingCache;
+import com.google.common.cache.RemovalListener;
+import com.google.common.cache.RemovalNotification;
+import com.google.common.collect.ImmutableList;
+import com.google.protobuf.Any;
+import com.google.protobuf.ByteString;
+import com.google.protobuf.InvalidProtocolBufferException;
+
+import io.grpc.Status;
+
+/**
+ * Proof of concept {@link FlightSqlProducer} implementation showing an Apache Derby backed Flight SQL server capable
+ * of the following workflows:
+ * - returning a list of tables from the action "GetTables".
+ * - creation of a prepared statement from the action "GetPreparedStatement".
+ * - execution of a prepared statement by using a {@link CommandPreparedStatementQuery} with getFlightInfo and
+ * getStream.
+ */
+public class FlightSqlExample extends FlightSqlProducer implements AutoCloseable {
+ private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(FlightSqlExample.class);
+
+ private static final int BATCH_ROW_SIZE = 1000;
+
+ private final Location location;
+ private final PoolingDataSource<PoolableConnection> dataSource;
+
+ private final LoadingCache<CommandPreparedStatementQuery, ResultSet> commandExecutePreparedStatementLoadingCache;
+ private final LoadingCache<PreparedStatementCacheKey, PreparedStatementContext> preparedStatementLoadingCache;
+
+ public FlightSqlExample(Location location) {
+ removeDerbyDatabaseIfExists();
+ populateDerbyDatabase();
+
+ final ConnectionFactory connectionFactory =
+ new DriverManagerConnectionFactory("jdbc:derby:target/derbyDB", null);
+ final PoolableConnectionFactory poolableConnectionFactory = new PoolableConnectionFactory(connectionFactory, null);
+ final ObjectPool<PoolableConnection> connectionPool = new GenericObjectPool<>(poolableConnectionFactory);
+ poolableConnectionFactory.setPool(connectionPool);
+
+ // PoolingDataSource takes ownership of connectionPool.
+ dataSource = new PoolingDataSource<>(connectionPool);
+
+ preparedStatementLoadingCache =
+ CacheBuilder.newBuilder()
+ .maximumSize(100)
+ .expireAfterWrite(10, java.util.concurrent.TimeUnit.MINUTES)
+ .removalListener(new PreparedStatementRemovalListener())
+ .build(new PreparedStatementCacheLoader(dataSource));
+
+ commandExecutePreparedStatementLoadingCache =
+ CacheBuilder.newBuilder()
+ .maximumSize(100)
+ .expireAfterWrite(10, java.util.concurrent.TimeUnit.MINUTES)
+ .removalListener(new CommandExecutePreparedStatementRemovalListener())
+ .build(new CommandExecutePreparedStatementCacheLoader(preparedStatementLoadingCache));
+
+ this.location = location;
+ }
+
+ @Override
+ public void getTables(FlightSql.ActionGetTablesRequest request, CallContext context,
+ StreamListener<Result> listener) {
+ try {
+ final String catalog = (request.getCatalog().isEmpty() ? null : request.getCatalog());
+
+ final String schemaFilterPattern =
+ (request.getSchemaFilterPattern().isEmpty() ? null : request.getSchemaFilterPattern());
+
+ final String tableFilterPattern =
+ (request.getTableNameFilterPattern().isEmpty() ? null : request.getTableNameFilterPattern());
+
+ final String[] tableTypes = request.getTableTypesList().size() == 0 ? null :
+ request.getTableTypesList().toArray(new String[request.getTableTypesList().size()]);
+
+ try (final Connection connection = dataSource.getConnection();
+ final ResultSet tables = connection.getMetaData().getTables(
+ catalog,
+ schemaFilterPattern,
+ tableFilterPattern,
+ tableTypes)) {
+ while (tables.next()) {
+ listener.onNext(getTableResult(tables, request.getIncludeSchema()));
+ }
+ }
+ } catch (SQLException e) {
+ listener.onError(e);
+ } finally {
+ listener.onCompleted();
+ }
+ }
+
+ private Result getTableResult(final ResultSet tables, boolean includeSchema) throws SQLException {
+
+ final String catalog = tables.getString("TABLE_CAT");
+ final String schema = tables.getString("TABLE_SCHEM");
+ final String table = tables.getString("TABLE_NAME");
+ final String tableType = tables.getString("TABLE_TYPE");
+
+ final ActionGetTablesResult.Builder builder = ActionGetTablesResult.newBuilder()
+ .setCatalog(catalog)
+ .setSchema(schema)
+ .setTable(table)
+ .setTableType(tableType);
+
+ if (includeSchema) {
+ final Schema pojoSchema = buildSchema(catalog, schema, table);
+ builder.setArrowMetadata(ByteString.copyFrom(pojoSchema.toByteArray()));
+ }
+
+ return new Result(Any.pack(builder.build()).toByteArray());
+ }
+
+ @Override
+ public void getPreparedStatement(FlightSql.ActionGetPreparedStatementRequest request, CallContext context,
+ StreamListener<Result> listener) {
+ final PreparedStatementCacheKey handle = new PreparedStatementCacheKey(
+ UUID.randomUUID().toString(), request.getQuery());
+
+ try {
+ final PreparedStatementContext preparedStatementContext = preparedStatementLoadingCache.get(handle);
+ final PreparedStatement preparedStatement = preparedStatementContext.getPreparedStatement();
+
+ // todo
+ final Schema pojoParameterMetaDataSchema = buildSchema(preparedStatement.getParameterMetaData());
+ final Schema pojoResultSetSchema = buildSchema(preparedStatement.getMetaData());
+
+ listener.onNext(new Result(
+ Any.pack(ActionGetPreparedStatementResult.newBuilder()
+ .setDatasetSchema(ByteString.copyFrom(pojoResultSetSchema.toByteArray()))
+ .setParameterSchema(ByteString.copyFrom(pojoParameterMetaDataSchema.toByteArray()))
+ .setPreparedStatementHandle(handle.toProtocol())
+ .build())
+ .toByteArray()));
+
+ } catch (ExecutionException | SQLException e) {
+ listener.onError(e);
+ } finally {
+ listener.onCompleted();
+ }
+ }
+
+ @Override
+ public FlightInfo getFlightInfoPreparedStatement(CommandPreparedStatementQuery command, FlightDescriptor descriptor,
+ CallContext context) {
+ try {
+ final ResultSet resultSet = commandExecutePreparedStatementLoadingCache.get(command);
+ final Schema schema = buildSchema(resultSet.getMetaData());
+
+ final List<FlightEndpoint> endpoints = ImmutableList
+ .of(new FlightEndpoint(new Ticket(Any.pack(command).toByteArray()), location));
+
+ return new FlightInfo(schema, descriptor, endpoints, -1, -1);
+ } catch (ExecutionException | SQLException e) {
+ logger.error("There was a problem executing the prepared statement", e);
+ throw new FlightRuntimeException(new CallStatus(FlightStatusCode.INTERNAL, e, e.getMessage(), null));
+ }
+ }
+
+ private Schema buildSchema(String catalog, String schema, String table) throws SQLException {
+ final List<Field> fields = new ArrayList<>();
+
+ try (final Connection connection = dataSource.getConnection();
+ final ResultSet columns = connection.getMetaData().getColumns(
+ catalog,
+ schema,
+ table,
+ null);) {
+
+ while (columns.next()) {
+ final String columnName = columns.getString("COLUMN_NAME");
+ final int jdbcDataType = columns.getInt("DATA_TYPE");
+ final String jdbcDataTypeName = columns.getString("TYPE_NAME");
+ final String jdbcIsNullable = columns.getString("IS_NULLABLE");
+ final boolean arrowIsNullable = jdbcIsNullable.equals("YES");
+
+ final int precision = columns.getInt("DECIMAL_DIGITS");
+ final int scale = columns.getInt("COLUMN_SIZE");
+ final ArrowType arrowType = FlightSqlUtils.getArrowTypeFromJDBCType(jdbcDataType, precision, scale);
+
+ final FieldType fieldType = new FieldType(arrowIsNullable, arrowType, /*dictionary=*/null);
+ fields.add(new Field(columnName, fieldType, null));
+ }
+ }
+
+ return new Schema(fields);
+ }
+
+ @Override
+ public void getStreamPreparedStatement(CommandPreparedStatementQuery command, CallContext context, Ticket ticket,
+ ServerStreamListener listener) {
+ try {
+ final ResultSet resultSet = commandExecutePreparedStatementLoadingCache.get(command);
+ final ResultSetMetaData resultSetMetaData = resultSet.getMetaData();
+ final Schema schema = buildSchema(resultSetMetaData);
+ final DictionaryProvider dictionaryProvider = new DictionaryProvider.MapDictionaryProvider();
+
+ try (final BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+ final VectorSchemaRoot root = VectorSchemaRoot.create(schema, allocator)) {
+
+ listener.start(root, dictionaryProvider);
+ final int columnCount = resultSetMetaData.getColumnCount();
+
+ while (resultSet.next()) {
+ final int rowCounter = readBatch(resultSet, resultSetMetaData, root, columnCount);
+
+ for (int resultSetColumnCounter = 1; resultSetColumnCounter <= columnCount; resultSetColumnCounter++) {
+ final String columnName = resultSetMetaData.getColumnName(resultSetColumnCounter);
+ root.getVector(columnName).setValueCount(rowCounter);
+ }
+
+ root.setRowCount(rowCounter);
+ listener.putNext();
+ }
+ }
+ } catch (ExecutionException | SQLException e) {
Review comment:
Need to catch RuntimeExceptions too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-871550608
@lidavidm - I think we've gotten agreement on everything pending the removal of `custom_info` - if so then I'll start work on updating the examples to adhere to the updated proto.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r634799405
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
Review comment:
Yes, these are modeled after ODBC/JDBC pattern value arguments which are the same as SQL LIKE patterns (minus the ability to override the LIKE escape sequence).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r567492176
##########
File path: java/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/PreparedStatementContext.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.util.Objects;
+
+import org.apache.arrow.util.AutoCloseables;
+
+class PreparedStatementContext implements AutoCloseable {
Review comment:
Also does this belong in a different java package? maybe jdbc?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r573039463
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
Review comment:
In the SQL approach define a table like:
`__APACHE_ARROW_FLIGHT_CATALOGS` minimal schema:
```
<catalog_name: utf8>
```
Supports project and optionally sorting on `catalog_name`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r634804879
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSQLClientUtils.java
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.arrow.flight.Action;
+import org.apache.arrow.flight.FlightClient;
+import org.apache.arrow.flight.FlightDescriptor;
+import org.apache.arrow.flight.FlightInfo;
+import org.apache.arrow.flight.Result;
+import org.apache.arrow.flight.sql.impl.FlightSQL;
+import org.apache.arrow.flight.sql.impl.FlightSQL.ActionGetPreparedStatementResult;
+import org.apache.arrow.flight.sql.impl.FlightSQL.ActionGetTablesRequest;
+import org.apache.arrow.flight.sql.impl.FlightSQL.ActionGetTablesResult;
+import org.apache.arrow.flight.sql.impl.FlightSQL.CommandPreparedStatementQuery;
+import org.apache.arrow.vector.types.pojo.Schema;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.ByteString;
+
+import io.grpc.Status;
+
+/**
+ * Client side utilities to work with Flight SQL semantics.
+ */
+public final class FlightSQLClientUtils {
Review comment:
I'm thinking instead of a utils class to represent client calls we make a wrapper FlightSQLClient class on top of FlightClient and make the various operations instance methods on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r567204874
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
Review comment:
nit: preference for Sql camel case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r573037004
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
Review comment:
I assume these are intended to be keys in the maps below? For some of them should they be promoted to first class concepts?
Per my comment above. I think I would have a preference for moving as much as possible to sql queries. so if we defined a special table like
`__APACHE_FLIGHT_SQL_INFO__` with a schema
```
<
flight_sql_server_name: utf8
flight_sql_server_version: utf8
flight_sql_server_arrow_version: utf8 (or maybe modeled as a struct)
flight_sql_info: map<utf8, dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32 >>
>
```
where the int32 bitmask would also have a defined extension metadata for a custom type of bitmask.
The minimal table requirement is that it supports projection:
i.e. only `select * from __APACHE_FLIGHT_SQL_INFO__` or `select FLIGHT_SQL_VERSION from __APACHE_FLIGHT_SQL_INFO__`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r572996266
##########
File path: java/flight/flight-sql/src/test/protobuf/flightSQLExample.proto
##########
@@ -0,0 +1,26 @@
+/*
Review comment:
Yes I this is why I didn't comment on file naming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r573028119
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
Review comment:
```suggestion
message ActionGetSqlInfoResult {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664923251
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSqlProducer.java
##########
@@ -0,0 +1,541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.arrow.flight.Action;
+import org.apache.arrow.flight.ActionType;
+import org.apache.arrow.flight.FlightDescriptor;
+import org.apache.arrow.flight.FlightInfo;
+import org.apache.arrow.flight.FlightProducer;
+import org.apache.arrow.flight.FlightStream;
+import org.apache.arrow.flight.PutResult;
+import org.apache.arrow.flight.Result;
+import org.apache.arrow.flight.SchemaResult;
+import org.apache.arrow.flight.Ticket;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionClosePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionCreatePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetCatalogs;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetSchemas;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetSqlInfo;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetTableTypes;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetTables;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementQuery;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementUpdate;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandStatementQuery;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandStatementUpdate;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.UnionMode;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.types.pojo.Schema;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+
+import io.grpc.Status;
+
+/**
+ * API to Implement an Arrow Flight SQL producer.
+ */
+public abstract class FlightSqlProducer implements FlightProducer, AutoCloseable {
+ /**
+ * Depending on the provided command, method either:
+ * 1. Return information about a SQL query, or
+ * 2. Return information about a prepared statement. In this case, parameters binding is allowed.
+ *
+ * @param context Per-call context.
+ * @param descriptor The descriptor identifying the data stream.
+ * @return information about the given SQL query, or the given prepared statement.
+ */
+ @Override
+ public FlightInfo getFlightInfo(CallContext context, FlightDescriptor descriptor) {
+ final Any command = FlightSqlUtils.parseOrThrow(descriptor.getCommand());
+
+ if (command.is(CommandStatementQuery.class)) {
+ return getFlightInfoStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandStatementQuery.class), context, descriptor);
+ } else if (command.is(CommandPreparedStatementQuery.class)) {
+ return getFlightInfoPreparedStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandPreparedStatementQuery.class), context, descriptor);
+ } else if (command.is(CommandGetCatalogs.class)) {
+ return getFlightInfoCatalogs(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetCatalogs.class), context, descriptor);
+ } else if (command.is(CommandGetSchemas.class)) {
+ return getFlightInfoSchemas(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetSchemas.class), context, descriptor);
+ } else if (command.is(CommandGetTables.class)) {
+ return getFlightInfoTables(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetTables.class), context, descriptor);
+ } else if (command.is(CommandGetTableTypes.class)) {
+ return getFlightInfoTableTypes(context, descriptor);
+ } else if (command.is(CommandGetSqlInfo.class)) {
+ return getFlightInfoSqlInfo(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetSqlInfo.class), context, descriptor);
+ }
+
+ throw Status.INVALID_ARGUMENT.asRuntimeException();
+ }
+
+ /**
+ * Returns the schema of the result produced by the SQL query.
+ *
+ * @param context Per-call context.
+ * @param descriptor The descriptor identifying the data stream.
+ * @return the result set schema.
+ */
+ @Override
+ public SchemaResult getSchema(CallContext context, FlightDescriptor descriptor) {
+ final Any command = FlightSqlUtils.parseOrThrow(descriptor.getCommand());
+
+ if (command.is(CommandStatementQuery.class)) {
+ return getSchemaStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandStatementQuery.class), context, descriptor);
+ } else if (command.is(CommandGetCatalogs.class)) {
+ return getSchemaCatalogs();
+ } else if (command.is(CommandGetSchemas.class)) {
+ return getSchemaSchemas();
+ } else if (command.is(CommandGetTables.class)) {
+ return getSchemaTables();
+ } else if (command.is(CommandGetTableTypes.class)) {
+ return getSchemaTableTypes();
+ } else if (command.is(CommandGetSqlInfo.class)) {
+ return getSchemaSqlInfo();
+ }
+
+ throw Status.INVALID_ARGUMENT.asRuntimeException();
+ }
+
+ /**
+ * Depending on the provided command, method either:
+ * 1. Return data for a stream produced by executing the provided SQL query, or
+ * 2. Return data for a prepared statement. In this case, parameters binding is allowed.
+ *
+ * @param context Per-call context.
+ * @param ticket The application-defined ticket identifying this stream.
+ * @param listener An interface for sending data back to the client.
+ */
+ @Override
+ public void getStream(CallContext context, Ticket ticket, ServerStreamListener listener) {
+ final Any command;
+
+ try {
+ command = Any.parseFrom(ticket.getBytes());
+ } catch (InvalidProtocolBufferException e) {
+ listener.error(e);
+ return;
+ }
+
+ if (command.is(CommandStatementQuery.class)) {
+ getStreamStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandStatementQuery.class), context, ticket, listener);
+ } else if (command.is(CommandPreparedStatementQuery.class)) {
+ getStreamPreparedStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandPreparedStatementQuery.class), context, ticket, listener);
+ } else if (command.is(CommandGetCatalogs.class)) {
+ getStreamCatalogs(context, ticket, listener);
+ } else if (command.is(CommandGetSchemas.class)) {
+ getStreamSchemas(FlightSqlUtils.unpackOrThrow(command, CommandGetSchemas.class), context, ticket, listener);
+ } else if (command.is(CommandGetTables.class)) {
+ getStreamTables(FlightSqlUtils.unpackOrThrow(command, CommandGetTables.class), context, ticket, listener);
+ } else if (command.is(CommandGetTableTypes.class)) {
+ getStreamTableTypes(context, ticket, listener);
+ } else if (command.is(CommandGetSqlInfo.class)) {
+ getStreamSqlInfo(FlightSqlUtils.unpackOrThrow(command, CommandGetSqlInfo.class), context, ticket, listener);
+ } else {
+ throw Status.INVALID_ARGUMENT.asRuntimeException();
Review comment:
I looked at the caller, and this is set on the listener as the caller wraps this already. I think it would be redundant to do so again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664911443
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSqlProducer.java
##########
@@ -0,0 +1,541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.arrow.flight.Action;
+import org.apache.arrow.flight.ActionType;
+import org.apache.arrow.flight.FlightDescriptor;
+import org.apache.arrow.flight.FlightInfo;
+import org.apache.arrow.flight.FlightProducer;
+import org.apache.arrow.flight.FlightStream;
+import org.apache.arrow.flight.PutResult;
+import org.apache.arrow.flight.Result;
+import org.apache.arrow.flight.SchemaResult;
+import org.apache.arrow.flight.Ticket;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionClosePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionCreatePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetCatalogs;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetSchemas;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetSqlInfo;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetTableTypes;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetTables;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementQuery;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementUpdate;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandStatementQuery;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandStatementUpdate;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.UnionMode;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.types.pojo.Schema;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+
+import io.grpc.Status;
+
+/**
+ * API to Implement an Arrow Flight SQL producer.
+ */
+public abstract class FlightSqlProducer implements FlightProducer, AutoCloseable {
+ /**
+ * Depending on the provided command, method either:
+ * 1. Return information about a SQL query, or
+ * 2. Return information about a prepared statement. In this case, parameters binding is allowed.
+ *
+ * @param context Per-call context.
+ * @param descriptor The descriptor identifying the data stream.
+ * @return information about the given SQL query, or the given prepared statement.
+ */
+ @Override
+ public FlightInfo getFlightInfo(CallContext context, FlightDescriptor descriptor) {
+ final Any command = FlightSqlUtils.parseOrThrow(descriptor.getCommand());
+
+ if (command.is(CommandStatementQuery.class)) {
+ return getFlightInfoStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandStatementQuery.class), context, descriptor);
+ } else if (command.is(CommandPreparedStatementQuery.class)) {
+ return getFlightInfoPreparedStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandPreparedStatementQuery.class), context, descriptor);
+ } else if (command.is(CommandGetCatalogs.class)) {
+ return getFlightInfoCatalogs(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetCatalogs.class), context, descriptor);
+ } else if (command.is(CommandGetSchemas.class)) {
+ return getFlightInfoSchemas(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetSchemas.class), context, descriptor);
+ } else if (command.is(CommandGetTables.class)) {
+ return getFlightInfoTables(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetTables.class), context, descriptor);
+ } else if (command.is(CommandGetTableTypes.class)) {
+ return getFlightInfoTableTypes(context, descriptor);
+ } else if (command.is(CommandGetSqlInfo.class)) {
+ return getFlightInfoSqlInfo(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetSqlInfo.class), context, descriptor);
+ }
+
+ throw Status.INVALID_ARGUMENT.asRuntimeException();
+ }
+
+ /**
+ * Returns the schema of the result produced by the SQL query.
+ *
+ * @param context Per-call context.
+ * @param descriptor The descriptor identifying the data stream.
+ * @return the result set schema.
+ */
+ @Override
+ public SchemaResult getSchema(CallContext context, FlightDescriptor descriptor) {
+ final Any command = FlightSqlUtils.parseOrThrow(descriptor.getCommand());
+
+ if (command.is(CommandStatementQuery.class)) {
+ return getSchemaStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandStatementQuery.class), context, descriptor);
+ } else if (command.is(CommandGetCatalogs.class)) {
+ return getSchemaCatalogs();
+ } else if (command.is(CommandGetSchemas.class)) {
+ return getSchemaSchemas();
+ } else if (command.is(CommandGetTables.class)) {
+ return getSchemaTables();
+ } else if (command.is(CommandGetTableTypes.class)) {
+ return getSchemaTableTypes();
+ } else if (command.is(CommandGetSqlInfo.class)) {
+ return getSchemaSqlInfo();
+ }
+
+ throw Status.INVALID_ARGUMENT.asRuntimeException();
+ }
+
+ /**
+ * Depending on the provided command, method either:
+ * 1. Return data for a stream produced by executing the provided SQL query, or
+ * 2. Return data for a prepared statement. In this case, parameters binding is allowed.
+ *
+ * @param context Per-call context.
+ * @param ticket The application-defined ticket identifying this stream.
+ * @param listener An interface for sending data back to the client.
+ */
+ @Override
+ public void getStream(CallContext context, Ticket ticket, ServerStreamListener listener) {
+ final Any command;
+
+ try {
+ command = Any.parseFrom(ticket.getBytes());
+ } catch (InvalidProtocolBufferException e) {
+ listener.error(e);
+ return;
+ }
+
+ if (command.is(CommandStatementQuery.class)) {
+ getStreamStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandStatementQuery.class), context, ticket, listener);
+ } else if (command.is(CommandPreparedStatementQuery.class)) {
+ getStreamPreparedStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandPreparedStatementQuery.class), context, ticket, listener);
+ } else if (command.is(CommandGetCatalogs.class)) {
+ getStreamCatalogs(context, ticket, listener);
+ } else if (command.is(CommandGetSchemas.class)) {
+ getStreamSchemas(FlightSqlUtils.unpackOrThrow(command, CommandGetSchemas.class), context, ticket, listener);
+ } else if (command.is(CommandGetTables.class)) {
+ getStreamTables(FlightSqlUtils.unpackOrThrow(command, CommandGetTables.class), context, ticket, listener);
+ } else if (command.is(CommandGetTableTypes.class)) {
+ getStreamTableTypes(context, ticket, listener);
+ } else if (command.is(CommandGetSqlInfo.class)) {
+ getStreamSqlInfo(FlightSqlUtils.unpackOrThrow(command, CommandGetSqlInfo.class), context, ticket, listener);
+ } else {
+ throw Status.INVALID_ARGUMENT.asRuntimeException();
+ }
+ }
+
+ /**
+ * Depending on the provided command, method either:
+ * 1. Execute provided SQL query as an update statement, or
+ * 2. Execute provided update SQL query prepared statement. In this case, parameters binding
+ * is allowed, or
+ * 3. Binds parameters to the provided prepared statement.
+ *
+ * @param context Per-call context.
+ * @param flightStream The data stream being uploaded.
+ * @param ackStream The data stream listener for update result acknowledgement.
+ * @return a Runnable to process the stream.
+ */
+ @Override
+ public Runnable acceptPut(CallContext context, FlightStream flightStream, StreamListener<PutResult> ackStream) {
+ final Any command = FlightSqlUtils.parseOrThrow(flightStream.getDescriptor().getCommand());
+
+ if (command.is(CommandStatementUpdate.class)) {
+ return acceptPutStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandStatementUpdate.class),
+ context, flightStream, ackStream);
+
+ } else if (command.is(CommandPreparedStatementUpdate.class)) {
+ return acceptPutPreparedStatementUpdate(
+ FlightSqlUtils.unpackOrThrow(command, CommandPreparedStatementUpdate.class),
+ context, flightStream, ackStream);
+
+ } else if (command.is(CommandPreparedStatementQuery.class)) {
+ return acceptPutPreparedStatementQuery(
+ FlightSqlUtils.unpackOrThrow(command, CommandPreparedStatementQuery.class),
+ context, flightStream, ackStream);
+ }
+
+ throw Status.INVALID_ARGUMENT.asRuntimeException();
Review comment:
The listener needs the error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-776084026
> Thinking about this a bit more I think an alternative to a lot of the metadata querying is to define a minimal sql table schema and querying capabilities for it. I'll expand more in a little bit
I did a rough sketch on how the metadata layer could be modeled as pseudo-columns as comments on the proto. I think this addresses some uneasiner I had with the existing model:
1. It uses one consistent data model for tabular data (the arrow model).
2. It makes query semantics clearer instead of trying to model them in an ad-hoc manner through protobufs
This comes at slightly more complex for server implementations. And potential confusion on limitations for specific tables. This might be limited to some extend if we had a different GetMetadata action so implementations know to expect a query to specific pseudo-table. I expect we would provide library wrappers over JDBC to also ease implementation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r567492009
##########
File path: java/flight/flight-sql/src/test/protobuf/flightSQLExample.proto
##########
@@ -0,0 +1,26 @@
+/*
Review comment:
nit: be consistent with file naming. Uppercase Flight
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r652026363
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
Review comment:
ODBC and JDBC define orderings for result sets like these, and I think it would likely be better to copy that rather than allow arbitrary ordering. I was trying to determine a use-case for when you'd want arbitrary ordering and wasn't able to think of a reasonable one for the additional effort required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664913704
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSqlUtils.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.sql.Types;
+import java.util.List;
+
+import org.apache.arrow.flight.ActionType;
+import org.apache.arrow.vector.types.DateUnit;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+
+import com.google.common.collect.ImmutableList;
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+import com.google.protobuf.Message;
+
+/**
+ * Utilities to work with Flight SQL semantics.
+ */
+public final class FlightSqlUtils {
+
+ private static final int BIT_WIDTH8 = 8;
+ private static final int BIT_WIDTH_16 = 16;
+ private static final int BIT_WIDTH_32 = 32;
+ private static final int BIT_WIDTH_64 = 64;
+ private static final boolean IS_SIGNED_FALSE = false;
+ private static final boolean IS_SIGNED_TRUE = true;
+
+ public static final ActionType FLIGHT_SQL_CREATEPREPAREDSTATEMENT = new ActionType("CreatePreparedStatement",
+ "Creates a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionCreatePreparedStatementRequest\n" +
+ "Response Message: ActionCreatePreparedStatementResult");
+
+ public static final ActionType FLIGHT_SQL_CLOSEPREPAREDSTATEMENT = new ActionType("ClosePreparedStatement",
+ "Closes a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionClosePreparedStatementRequest\n" +
+ "Response Message: N/A");
+
+ public static final List<ActionType> FLIGHT_SQL_ACTIONS = ImmutableList.of(
+ FLIGHT_SQL_CREATEPREPAREDSTATEMENT,
+ FLIGHT_SQL_CLOSEPREPAREDSTATEMENT
+ );
+
+ /**
+ * Converts {@link java.sql.Types} values returned from JDBC Apis to Arrow types.
+ *
+ * @param jdbcDataType {@link java.sql.Types} value.
+ * @param precision Precision of the type.
+ * @param scale Scale of the type.
+ * @return The Arrow equivalent type.
+ */
+ public static ArrowType getArrowTypeFromJDBCType(int jdbcDataType, int precision, int scale) {
+
+ switch (jdbcDataType) {
+ case Types.BIT:
+ case Types.BOOLEAN:
+ return ArrowType.Bool.INSTANCE;
+ case Types.TINYINT:
+ return new ArrowType.Int(BIT_WIDTH8, IS_SIGNED_TRUE);
+ case Types.SMALLINT:
+ return new ArrowType.Int(BIT_WIDTH_16, IS_SIGNED_TRUE);
+ case Types.INTEGER:
+ return new ArrowType.Int(BIT_WIDTH_32, IS_SIGNED_TRUE);
+ case Types.BIGINT:
+ return new ArrowType.Int(BIT_WIDTH_64, IS_SIGNED_TRUE);
+ case Types.FLOAT:
+ case Types.REAL:
+ return new ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE);
+ case Types.DOUBLE:
+ return new ArrowType.FloatingPoint(FloatingPointPrecision.DOUBLE);
+ case Types.NUMERIC:
+ case Types.DECIMAL:
+ return new ArrowType.Decimal(precision, scale);
+ case Types.DATE:
+ return new ArrowType.Date(DateUnit.DAY);
+ case Types.TIME:
+ return new ArrowType.Time(TimeUnit.MILLISECOND, BIT_WIDTH_32);
+ case Types.TIMESTAMP:
+ return new ArrowType.Timestamp(TimeUnit.MILLISECOND, null);
+ case Types.BINARY:
+ case Types.VARBINARY:
+ case Types.LONGVARBINARY:
+ return ArrowType.Binary.INSTANCE;
+ case Types.NULL:
+ return ArrowType.Null.INSTANCE;
+
+ case Types.CHAR:
+ case Types.VARCHAR:
+ case Types.LONGVARCHAR:
+ case Types.CLOB:
+ case Types.NCHAR:
+ case Types.NVARCHAR:
+ case Types.LONGNVARCHAR:
+ case Types.NCLOB:
+
+ case Types.OTHER:
+ case Types.JAVA_OBJECT:
+ case Types.DISTINCT:
+ case Types.STRUCT:
Review comment:
Should be an Arrow struct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r573037004
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
Review comment:
I assume these are intended to be keys in the maps below? For some of them should they be promoted to first class concepts?
Per my comment above. I think I would have a preference for moving as much as possible to sql queries. so if we defined a special table like
`__APACHE_FLIGHT_SQL_INFO__` with a schema
{
flight_sql_server_name: utf8
flight_sql_server_version: utf8
flight_sql_server_arrow_version: utf8 (or maybe modeled as a struct)
flight_sql_info: map<utf8, dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32 >>
}
where the int32 bitmask would also have a defined extension metadata for a custom type of bitmask.
The minimal table requirement is that it supports projection:
i.e. only `select * from __APACHE_FLIGHT_SQL_INFO__` or `select FLIGHT_SQL_VERSION from __APACHE_FLIGHT_SQL_INFO__`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-776089540
> We don't have a GetMetadata action, but we could encode it in GetFlightInfo by sending a different command payload. That would let us cleanly namespace the special tables and the actual tables.
Sorry need to reacquaint myself with all the flight specific nomenclature, this sounds like what I was thinking.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r635405341
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
Review comment:
> I think just submitting metadata requests via SQL has other challenges from an implementation perspective. It necessitates either having the backend validate parsing of the pseudo table (essentially making it a real table/view) or having the FlightProducer parse the query, identify the pseudo table and its filters, then translating the request to a backend call.
I agree it can be complicated, I think in many cases vendors already support this, and JDBC drivers end up doing this under the hood. I haven't looked deeply but I was hoping this type of parsing would not be hard with something like Calcite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r660888081
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
Review comment:
Updated to use the existing DoGet mechanism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-870840510
@lidavidm - the proto has been updated (nothing else) to reflect what I think is the most recent understanding - if you want to take a look and let me know if there are any other thoughts. Will update the actual examples after the fact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] bobtins commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
bobtins commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r614408197
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
Review comment:
precedence (third time's the charm :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r567492063
##########
File path: java/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/PreparedStatementContext.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.util.Objects;
+
+import org.apache.arrow.util.AutoCloseables;
+
+class PreparedStatementContext implements AutoCloseable {
Review comment:
javadoc please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-776086029
> This comes at slightly more complex for server implementations. And potential confusion on limitations for specific tables. This might be limited to some extend if we had a different GetMetadata action so implementations know to expect a query to specific pseudo-table. I expect we would provide library wrappers over JDBC to also ease implementation.
We don't have a GetMetadata action, but we could encode it in GetFlightInfo by sending a different command payload. That would let us cleanly namespace the special tables and the actual tables.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r635420995
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
Review comment:
Yes, it'd be doable with Calcite. The gotchas to me are:
- Having the FlightSQLProducer parse to find pseudo-tables must be optional.
- We can provide a partial implementation of this that invokes Calcite's parser, but it's the implementer's responsibility to configure Calcite in a way that matches their SQL backend's syntax (eg identifier quote characters, LIKE escape sequences).
- We fallback silently if the table used is a non-Flight table. There should be nothing indicating a possible error since this is the primary use case.
- If the implementer uses the pre-built pseudo-table parser, its filtering and table transformation capabilities are limited to what similar ODBC/JDBC catalog functions can do (eg can't re-order columns, invoke scalar/aggregate functions or arithmetic expressions, pattern value filtering on only certain columns)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r635534750
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSQLClientUtils.java
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.arrow.flight.Action;
+import org.apache.arrow.flight.FlightClient;
+import org.apache.arrow.flight.FlightDescriptor;
+import org.apache.arrow.flight.FlightInfo;
+import org.apache.arrow.flight.Result;
+import org.apache.arrow.flight.sql.impl.FlightSQL;
+import org.apache.arrow.flight.sql.impl.FlightSQL.ActionGetPreparedStatementResult;
+import org.apache.arrow.flight.sql.impl.FlightSQL.ActionGetTablesRequest;
+import org.apache.arrow.flight.sql.impl.FlightSQL.ActionGetTablesResult;
+import org.apache.arrow.flight.sql.impl.FlightSQL.CommandPreparedStatementQuery;
+import org.apache.arrow.vector.types.pojo.Schema;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.ByteString;
+
+import io.grpc.Status;
+
+/**
+ * Client side utilities to work with Flight SQL semantics.
+ */
+public final class FlightSQLClientUtils {
+
+ /**
+ * Helper method to request a list of tables from a Flight SQL enabled endpoint.
+ *
+ * @param client The Flight Client.
+ * @param catalog The catalog.
+ * @param schemaFilterPattern The schema filter pattern.
+ * @param tableFilterPattern The table filter pattern.
+ * @param tableTypes The table types to include.
+ * @param includeSchema True to include the schema upon return, false to not include the schema.
+ * @return A list of tables matching the criteria.
+ */
+ public static List<ActionGetTablesResult> getTables(FlightClient client, String catalog, String schemaFilterPattern,
+ String tableFilterPattern, List<String> tableTypes, boolean includeSchema) {
+
+ final ActionGetTablesRequest.Builder requestBuilder = ActionGetTablesRequest
+ .newBuilder()
+ .setIncludeSchema(includeSchema);
+
+ if (catalog != null) {
+ requestBuilder.setCatalog(catalog);
+ }
+
+ if (schemaFilterPattern != null) {
+ requestBuilder.setSchemaFilterPattern(schemaFilterPattern);
+ }
+
+ if (tableFilterPattern != null) {
+ requestBuilder.setTableNameFilterPattern(tableFilterPattern);
+ }
+
+ if (tableTypes != null) {
+ requestBuilder.addAllTableTypes(tableTypes);
+ }
+
+ final Iterator<Result> results = client.doAction(new Action(
+ "GetTables", Any.pack(requestBuilder.build()).toByteArray()));
+
+ final List<ActionGetTablesResult> getTablesResults = new ArrayList<>();
+ results.forEachRemaining(result -> {
+ ActionGetTablesResult actual = FlightSQLUtils.unpackAndParseOrThrow(result.getBody(),
+ ActionGetTablesResult.class);
+ getTablesResults.add(actual);
+ });
+
+ return getTablesResults;
+ }
+
+ /**
+ * Helper method to create a prepared statement on the server.
+ *
+ * @param client The Flight Client.
+ * @param query The query to prepare.
+ * @return Metadata and handles to the prepared statement which exists on the server.
+ */
+ public static FlightSQLPreparedStatement getPreparedStatement(FlightClient client, String query) {
+ return new FlightSQLPreparedStatement(client, query);
+ }
+
+ /**
+ * Helper class to encapsulate Flight SQL prepared statement logic.
+ */
+ public static class FlightSQLPreparedStatement implements Closeable {
+ private final FlightClient client;
+ private final ActionGetPreparedStatementResult preparedStatementResult;
+ private long invocationCount;
+ private boolean isClosed;
+ private Schema resultSetSchema = null;
+ private Schema parameterSchema = null;
+
+ /**
+ * Constructor.
+ *
+ * @param client The client. FlightSQLPreparedStatement does not maintain this resource.
+ * @param sql The query.
+ */
+ public FlightSQLPreparedStatement(FlightClient client, String sql) {
+ this.client = client;
+
+ final Iterator<Result> preparedStatementResults = client.doAction(new Action("GetPreparedStatement",
+ Any.pack(FlightSQL.ActionGetPreparedStatementRequest
+ .newBuilder()
+ .setQuery(sql)
+ .build())
+ .toByteArray()));
+
+ preparedStatementResult = FlightSQLUtils.unpackAndParseOrThrow(
+ preparedStatementResults.next().getBody(),
+ ActionGetPreparedStatementResult.class);
+
+ invocationCount = 0;
+ isClosed = false;
+ }
+
+ /**
+ * Returns the Schema of the resultset.
+ *
+ * @return the Schema of the resultset.
+ */
+ public Schema getResultSetSchema() {
+ if (resultSetSchema == null && preparedStatementResult.getDatasetSchema() != null) {
+ resultSetSchema = Schema.deserialize(preparedStatementResult.getDatasetSchema().asReadOnlyByteBuffer());
+ }
+ return resultSetSchema;
+ }
+
+ /**
+ * Returns the Schema of the parameters.
+ *
+ * @return the Schema of the parameters.
+ */
+ public Schema getParameterSchema() {
+ if (parameterSchema == null && preparedStatementResult.getParameterSchema() != null) {
+ parameterSchema = Schema.deserialize(preparedStatementResult.getParameterSchema().asReadOnlyByteBuffer());
+ }
+ return parameterSchema;
+ }
+
+ /**
+ * Executes the prepared statement query on the server.
+ *
+ * @return a FlightInfo object representing the stream(s) to fetch.
+ * @throws IOException if the PreparedStatement is closed.
+ */
+ public FlightInfo executeQuery() throws IOException {
+ if (isClosed) {
+ throw new IOException("Prepared statement has already been closed on the server.");
+ }
+
+ final FlightDescriptor descriptor = FlightDescriptor
+ .command(Any.pack(CommandPreparedStatementQuery.newBuilder()
+ .setClientExecutionHandle(
+ ByteString.copyFrom(ByteBuffer.allocate(Long.BYTES).putLong(invocationCount++)))
+ .setPreparedStatementHandle(preparedStatementResult.getPreparedStatementHandle())
+ .build())
+ .toByteArray());
+
+ return client.getInfo(descriptor);
+ }
+
+ /**
+ * Executes the prepared statement update on the server.
+ *
+ * @return the number of rows updated.
+ */
+ public long executeUpdate() {
+ throw Status.UNIMPLEMENTED.asRuntimeException();
+ }
+
+ @Override
+ public void close() {
+ isClosed = true;
+ final Iterator<Result> closePreparedStatementResults = client.doAction(new Action("ClosePreparedStatement",
Review comment:
Multiple calls to close() should behave idempotently. We shouldn't re-issue the close request if close() was already called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r660926104
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
+
+ // Implementation specific custom SqlInfo values.
+ repeated string custom_info = 2;
+}
+
+/*
+ Represents a request to retrieve the list of catalogs on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name.
+ */
+message CommandGetCatalogs {
+}
+
+/*
+ * Represents a request to retrieve the list of schemas on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8
+ * >
+ * The returned data should be ordered by catalog_name, then schema_name.
+ */
+message CommandGetSchemas {
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then all catalogs are searched.
+ */
+ string catalog = 1;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 2;
+}
+
+/*
+ * Represents a request to retrieve the list of tables, and optionally their schemas, on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * catalog_name: utf8,
+ * schema_name: utf8,
+ * table_name: utf8,
+ * table_type: utf8,
+ * table_schema: bytes
+ * >
+ * The returned data should be ordered by catalog_name, schema_name, table_name, then table_type.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then all catalogs are searched.
+ */
+ string catalog = 1;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, all schemas matching other filters are searched.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 2;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no table_name_filter_pattern is provided, all tables matching other filters are searched.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 3;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 4;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 5;
+}
+
+/*
+ * Represents a request to retrieve the list of table types on a Flight SQL enabled backend.
+ * Used in the command member of FlightDescriptor for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * table_type: utf8
+ * >
+ * The returned data should be ordered by table_type.
+ */
+message CommandGetTableTypes {
+}
Review comment:
No - most are, but for example the prepared statement one is `DoPut` and `GetFlighTInfo`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664935458
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSqlUtils.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.sql.Types;
+import java.util.List;
+
+import org.apache.arrow.flight.ActionType;
+import org.apache.arrow.vector.types.DateUnit;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+
+import com.google.common.collect.ImmutableList;
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+import com.google.protobuf.Message;
+
+/**
+ * Utilities to work with Flight SQL semantics.
+ */
+public final class FlightSqlUtils {
+
+ private static final int BIT_WIDTH8 = 8;
+ private static final int BIT_WIDTH_16 = 16;
+ private static final int BIT_WIDTH_32 = 32;
+ private static final int BIT_WIDTH_64 = 64;
+ private static final boolean IS_SIGNED_FALSE = false;
+ private static final boolean IS_SIGNED_TRUE = true;
+
+ public static final ActionType FLIGHT_SQL_CREATEPREPAREDSTATEMENT = new ActionType("CreatePreparedStatement",
+ "Creates a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionCreatePreparedStatementRequest\n" +
+ "Response Message: ActionCreatePreparedStatementResult");
+
+ public static final ActionType FLIGHT_SQL_CLOSEPREPAREDSTATEMENT = new ActionType("ClosePreparedStatement",
+ "Closes a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionClosePreparedStatementRequest\n" +
+ "Response Message: N/A");
+
+ public static final List<ActionType> FLIGHT_SQL_ACTIONS = ImmutableList.of(
+ FLIGHT_SQL_CREATEPREPAREDSTATEMENT,
+ FLIGHT_SQL_CLOSEPREPAREDSTATEMENT
+ );
+
+ /**
+ * Converts {@link java.sql.Types} values returned from JDBC Apis to Arrow types.
+ *
+ * @param jdbcDataType {@link java.sql.Types} value.
+ * @param precision Precision of the type.
+ * @param scale Scale of the type.
+ * @return The Arrow equivalent type.
+ */
+ public static ArrowType getArrowTypeFromJDBCType(int jdbcDataType, int precision, int scale) {
+
+ switch (jdbcDataType) {
+ case Types.BIT:
+ case Types.BOOLEAN:
+ return ArrowType.Bool.INSTANCE;
+ case Types.TINYINT:
+ return new ArrowType.Int(BIT_WIDTH8, IS_SIGNED_TRUE);
+ case Types.SMALLINT:
+ return new ArrowType.Int(BIT_WIDTH_16, IS_SIGNED_TRUE);
+ case Types.INTEGER:
+ return new ArrowType.Int(BIT_WIDTH_32, IS_SIGNED_TRUE);
+ case Types.BIGINT:
+ return new ArrowType.Int(BIT_WIDTH_64, IS_SIGNED_TRUE);
+ case Types.FLOAT:
+ case Types.REAL:
+ return new ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE);
+ case Types.DOUBLE:
+ return new ArrowType.FloatingPoint(FloatingPointPrecision.DOUBLE);
+ case Types.NUMERIC:
+ case Types.DECIMAL:
+ return new ArrowType.Decimal(precision, scale);
+ case Types.DATE:
+ return new ArrowType.Date(DateUnit.DAY);
+ case Types.TIME:
+ return new ArrowType.Time(TimeUnit.MILLISECOND, BIT_WIDTH_32);
+ case Types.TIMESTAMP:
+ return new ArrowType.Timestamp(TimeUnit.MILLISECOND, null);
+ case Types.BINARY:
+ case Types.VARBINARY:
+ case Types.LONGVARBINARY:
+ return ArrowType.Binary.INSTANCE;
+ case Types.NULL:
+ return ArrowType.Null.INSTANCE;
+
+ case Types.CHAR:
+ case Types.VARCHAR:
+ case Types.LONGVARCHAR:
+ case Types.CLOB:
+ case Types.NCHAR:
+ case Types.NVARCHAR:
+ case Types.LONGNVARCHAR:
+ case Types.NCLOB:
+
+ case Types.OTHER:
+ case Types.JAVA_OBJECT:
+ case Types.DISTINCT:
+ case Types.STRUCT:
+ case Types.ARRAY:
+ case Types.BLOB:
+ case Types.REF:
+ case Types.DATALINK:
+ case Types.ROWID:
+ case Types.SQLXML:
+ case Types.REF_CURSOR:
+ case Types.TIME_WITH_TIMEZONE:
+ case Types.TIMESTAMP_WITH_TIMEZONE:
+ default:
+ return ArrowType.Utf8.INSTANCE;
+ }
+ }
+
+ /**
+ * Helper to parse {@link com.google.protobuf.Any} objects to the specific protobuf object.
+ *
+ * @param source the raw bytes source value.
+ * @return the materialized protobuf object.
+ */
+ public static Any parseOrThrow(byte[] source) {
Review comment:
I don't think so, taking into account the other comments about the callers already handling this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r573047556
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
Review comment:
SQL first approach:
`__APACHE_ARROW_FLIGHT_TABLES`
```
<
catalog: utf8
schema: utf8
table_type: utf8 # Implementations are recommended to dictionary encode this field
table: utf8
serialized_flatbuffer_table_schema: bytes
>
```
Table supports equality and like predicates on schema and table. Queries must specify catalog with an equality predicate. Support projection and ordering. Ordering might be limited to requiring field catalog, schema, table_type, table in that order.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#issuecomment-875670683
Just FYI: most CI pipelines will skip so long as you have the WIP in the PR title, feel free to remove that (keeping it in Draft if desired) when you're ready for CI to run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r661632984
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
Review comment:
Sounds good to me. If collisions are potentially a concern we could say something like all custom info must have some prefix that FlightSQL will not use.
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
Review comment:
Sounds good to me. If collisions are potentially a concern we could say something like all custom info reported by an implementation must have some prefix that FlightSQL will not use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r573037004
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
Review comment:
I assume these are intended to be keys in the maps below? For some of them should they be promoted to first class concepts?
Per my comment above. I think I would have a preference for moving as much as possible to sql queries. so if we defined a special table like
"__APACHE_FLIGHT_SQL_INFO__" with a schema
{
flight_sql_server_name: utf8
flight_sql_server_version: utf8
flight_sql_server_arrow_version: utf8 (or maybe modeled as a struct)
flight_sql_info: map<utf8, dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32 >>
}
where the int32 bitmask would also have a defined extension metadata for a custom type of bitmask.
The minimal table requirement is that it supports projection:
i.e. only `select * from __APACHE_FLIGHT_SQL_INFO__` or `select FLIGHT_SQL_VERSION from __APACHE_FLIGHT_SQL_INFO__`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r654543777
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 6;
+}
+
+/*
+ * Wrap the result of a "GetTables" action.
+ */
+message ActionGetTablesResult {
+ string catalog = 1;
+ string schema = 2;
+ string table = 3;
+ string table_type = 4;
+
+ /*
+ * Schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ * Null if include_schema on request is false.
+ */
+ bytes arrow_metadata = 5;
+}
+
+/*
+ * Wrap the result of a "GetTableTypes" action.
+ */
+message ActionGetTableTypesResult {
+ /*
+ * Indicates the type of the table. E.g. table (regular data table) , view, system table etc.
+ */
+ string table_type = 1;
+}
+
+// SQL Execution Action Messages
+
+/*
+ * Request message for the "GetPreparedStatement" action on a
+ * Flight SQL enabled backend.
+ */
+message ActionGetPreparedStatementRequest {
+ // The valid SQL string to get a prepared statement for.
+ string query = 1;
+}
+
+/*
+ * Wrap the result of a "GetPreparedStatement" action.
+ */
+message ActionGetPreparedStatementResult {
+
+ // Opaque handle for the prepared statement on the server.
+ bytes prepared_statement_handle = 1;
+
+ // If a result set generating query was provided, dataset_schema contains the
+ // schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ bytes dataset_schema = 2;
+
+ // If the query provided contained parameters, parameter_schema contains the
+ // Schema of the expected parameters as described in Schema.fbs::Schema.
+ bytes parameter_schema = 3;
+}
+
+/*
+ * Request message for the "ClosePreparedStatement" action on a
+ * Flight SQL enabled backend.
+ * Closes server resources associated with the prepared statement handle.
+ */
+message ActionClosePreparedStatementRequest {
+ // Opaque handle for the prepared statement on the server.
+ string prepared_statement_handle = 1;
+}
+
+
+// SQL Execution Messages.
+
+/*
+ * Represents a SQL query. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ */
+message CommandStatementQuery {
+ // The SQL syntax.
+ string query = 1;
+}
+
+/*
+ * Represents an instance of executing a prepared statement. Used in the
+ * command member of FlightDescriptor for the following RPC calls:
+ * TODO: Is the idea that a Put with parameter values would execute multiple bound versions of the prepared statement
Review comment:
Agreed that if we could change schema we could have this streamed back normally, but I'm not sure how far along that is.
I think, given the current design, I'd propose adding another command that's issued after the GetFlightInfo but before the DoGet to retrieve the parameter data, if any. I think this is the last bit to work out, if we agree on this I'll update the PR with the changes to reflect the current state and then we can update the examples around that.
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 6;
+}
+
+/*
+ * Wrap the result of a "GetTables" action.
+ */
+message ActionGetTablesResult {
+ string catalog = 1;
+ string schema = 2;
+ string table = 3;
+ string table_type = 4;
+
+ /*
+ * Schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ * Null if include_schema on request is false.
+ */
+ bytes arrow_metadata = 5;
+}
+
+/*
+ * Wrap the result of a "GetTableTypes" action.
+ */
+message ActionGetTableTypesResult {
+ /*
+ * Indicates the type of the table. E.g. table (regular data table) , view, system table etc.
+ */
+ string table_type = 1;
+}
+
+// SQL Execution Action Messages
+
+/*
+ * Request message for the "GetPreparedStatement" action on a
+ * Flight SQL enabled backend.
+ */
+message ActionGetPreparedStatementRequest {
+ // The valid SQL string to get a prepared statement for.
+ string query = 1;
+}
+
+/*
+ * Wrap the result of a "GetPreparedStatement" action.
+ */
+message ActionGetPreparedStatementResult {
+
+ // Opaque handle for the prepared statement on the server.
+ bytes prepared_statement_handle = 1;
+
+ // If a result set generating query was provided, dataset_schema contains the
+ // schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ bytes dataset_schema = 2;
+
+ // If the query provided contained parameters, parameter_schema contains the
+ // Schema of the expected parameters as described in Schema.fbs::Schema.
+ bytes parameter_schema = 3;
+}
+
+/*
+ * Request message for the "ClosePreparedStatement" action on a
+ * Flight SQL enabled backend.
+ * Closes server resources associated with the prepared statement handle.
+ */
+message ActionClosePreparedStatementRequest {
+ // Opaque handle for the prepared statement on the server.
+ string prepared_statement_handle = 1;
+}
+
+
+// SQL Execution Messages.
+
+/*
+ * Represents a SQL query. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ */
+message CommandStatementQuery {
+ // The SQL syntax.
+ string query = 1;
+}
+
+/*
+ * Represents an instance of executing a prepared statement. Used in the
+ * command member of FlightDescriptor for the following RPC calls:
+ * TODO: Is the idea that a Put with parameter values would execute multiple bound versions of the prepared statement
Review comment:
Yes - although we could also say it's after the DoGet as well if that simplified things. I'll work on updating this exempting the output parameters then so we can get this moving meaningfully again.
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
Review comment:
No - it has separate tables: TABLES and COLUMNS IIRC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r572960572
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
Review comment:
I think these are standard are also standard matches for like expressions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r572952134
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
Review comment:
Ah, according to the doc, it comes from JDBC. Might be worth noting at least once.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
jduo commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664913437
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSqlUtils.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.sql.Types;
+import java.util.List;
+
+import org.apache.arrow.flight.ActionType;
+import org.apache.arrow.vector.types.DateUnit;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+
+import com.google.common.collect.ImmutableList;
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+import com.google.protobuf.Message;
+
+/**
+ * Utilities to work with Flight SQL semantics.
+ */
+public final class FlightSqlUtils {
+
+ private static final int BIT_WIDTH8 = 8;
+ private static final int BIT_WIDTH_16 = 16;
+ private static final int BIT_WIDTH_32 = 32;
+ private static final int BIT_WIDTH_64 = 64;
+ private static final boolean IS_SIGNED_FALSE = false;
+ private static final boolean IS_SIGNED_TRUE = true;
+
+ public static final ActionType FLIGHT_SQL_CREATEPREPAREDSTATEMENT = new ActionType("CreatePreparedStatement",
+ "Creates a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionCreatePreparedStatementRequest\n" +
+ "Response Message: ActionCreatePreparedStatementResult");
+
+ public static final ActionType FLIGHT_SQL_CLOSEPREPAREDSTATEMENT = new ActionType("ClosePreparedStatement",
+ "Closes a reusable prepared statement resource on the server. \n" +
+ "Request Message: ActionClosePreparedStatementRequest\n" +
+ "Response Message: N/A");
+
+ public static final List<ActionType> FLIGHT_SQL_ACTIONS = ImmutableList.of(
+ FLIGHT_SQL_CREATEPREPAREDSTATEMENT,
+ FLIGHT_SQL_CLOSEPREPAREDSTATEMENT
+ );
+
+ /**
+ * Converts {@link java.sql.Types} values returned from JDBC Apis to Arrow types.
+ *
+ * @param jdbcDataType {@link java.sql.Types} value.
+ * @param precision Precision of the type.
+ * @param scale Scale of the type.
+ * @return The Arrow equivalent type.
+ */
+ public static ArrowType getArrowTypeFromJDBCType(int jdbcDataType, int precision, int scale) {
+
+ switch (jdbcDataType) {
+ case Types.BIT:
+ case Types.BOOLEAN:
+ return ArrowType.Bool.INSTANCE;
+ case Types.TINYINT:
+ return new ArrowType.Int(BIT_WIDTH8, IS_SIGNED_TRUE);
+ case Types.SMALLINT:
+ return new ArrowType.Int(BIT_WIDTH_16, IS_SIGNED_TRUE);
+ case Types.INTEGER:
+ return new ArrowType.Int(BIT_WIDTH_32, IS_SIGNED_TRUE);
+ case Types.BIGINT:
+ return new ArrowType.Int(BIT_WIDTH_64, IS_SIGNED_TRUE);
+ case Types.FLOAT:
+ case Types.REAL:
+ return new ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE);
+ case Types.DOUBLE:
+ return new ArrowType.FloatingPoint(FloatingPointPrecision.DOUBLE);
+ case Types.NUMERIC:
+ case Types.DECIMAL:
+ return new ArrowType.Decimal(precision, scale);
+ case Types.DATE:
+ return new ArrowType.Date(DateUnit.DAY);
+ case Types.TIME:
+ return new ArrowType.Time(TimeUnit.MILLISECOND, BIT_WIDTH_32);
Review comment:
Seems like these should be based on the number of fractional seconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r658861293
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
Review comment:
By 'RPC protocol' do you mean the version of Flight and of FlightSQL? That does make sense.
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
Review comment:
If we're going to return data as an Arrow record batch (I agree this makes sense) we should probably make all this data fetched through DoGet or DoExchange so that we're not specifying a separate encoding of Arrow data into Flight.
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
Review comment:
We should also specify the expected schema inline here.
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Wrapper for the result of a "GetSQLInfo" action.
+ */
+message ActionGetSQLInfoResult {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * -
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * Indicates whether the Flight SQL Server supports both read and write.
+ * FLIGHT_SQL_SERVER_READ_WRITE
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ map<string, GetSQLInfoValue> flight_sql_info = 1;
+
+ // Implementation specific custom SQLInfo values.
+ map<string, GetSQLInfoValue> custom_info = 2;
+}
+
+/*
+ * Wrapper for values returned in ActionGetSQLInfoResult.
+ */
+message GetSQLInfoValue {
+ oneof value {
+ string string_value = 1;
+ int32 integer_value = 2;
+ int32 integer_bitmask = 3;
+ int32 integer_flag = 4;
+ bytes binary_value = 5;
+ int64 long_value = 6;
+ }
+}
+
+/*
+ * Request message for the "GetCatalogs" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of catalogs available in the server.
+ */
+message ActionGetCatalogsRequest {
+ /*
+ * Specifies the order of result values.
+ */
+ ResultsOrder order = 1;
+}
+
+/*
+ * Wrap the result of a "GetCatalogs" action.
+ */
+message ActionGetCatalogsResult {
+ repeated string catalog = 1;
+}
+
+/*
+ * Request message for the "GetSchemas" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of schemas available in the server.
+ */
+message ActionGetSchemasRequest {
+ /*
+ * Specifies the order of result values with precedence:
+ * - catalog
+ * - schema
+ */
+ ResultsOrder order = 1;
+
+ /*
+ * Specifies the Catalog to search for schemas.
+ * If omitted, then schemas for all catalogs are searched.
+ */
+ string catalog = 2;
+
+ /*
+ * Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+}
+
+/*
+ * Wrap the result of a "GetSchemas" action.
+ */
+message ActionGetSchemasResult {
+ string catalog = 1;
+ string schema = 2;
+}
+
+/*
+ * Request message for the "GetTables" action on a
+ * Flight SQL enabled backend.
+ * Requests a list of tables available in the server.
+ */
+message ActionGetTablesRequest {
+ /*
+ * Specifies the order of result values with prescendence:
+ * - catalog
+ * - schema
+ * - table_type
+ * - table
+ */
+ ResultsOrder order = 1;
+
+ // Specifies the Catalog to search for schemas.
+ string catalog = 2;
+
+ /* Specifies a filter pattern for schemas to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string schema_filter_pattern = 3;
+
+ /* Specifies a filter pattern for tables to search for.
+ * When no schema_filter_pattern is provided, the pattern will not be used to narrow the search.
+ * In the pattern string, two special characters can be used to denote matching rules:
+ * - "%" means to match any substring with 0 or more characters.
+ * - "_" means to match any one character.
+ */
+ string table_name_filter_pattern = 4;
+
+ // Specifies a filter of table types which must match.
+ repeated string table_types = 5;
+
+ // Specifies if the schema should be returned for found tables.
+ bool include_schema = 6;
+}
+
+/*
+ * Wrap the result of a "GetTables" action.
+ */
+message ActionGetTablesResult {
+ string catalog = 1;
+ string schema = 2;
+ string table = 3;
+ string table_type = 4;
+
+ /*
+ * Schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ * Null if include_schema on request is false.
+ */
+ bytes arrow_metadata = 5;
+}
+
+/*
+ * Wrap the result of a "GetTableTypes" action.
+ */
+message ActionGetTableTypesResult {
+ /*
+ * Indicates the type of the table. E.g. table (regular data table) , view, system table etc.
+ */
+ string table_type = 1;
+}
+
+// SQL Execution Action Messages
+
+/*
+ * Request message for the "GetPreparedStatement" action on a
+ * Flight SQL enabled backend.
+ */
+message ActionGetPreparedStatementRequest {
+ // The valid SQL string to get a prepared statement for.
+ string query = 1;
+}
+
+/*
+ * Wrap the result of a "GetPreparedStatement" action.
+ */
+message ActionGetPreparedStatementResult {
+
+ // Opaque handle for the prepared statement on the server.
+ bytes prepared_statement_handle = 1;
+
+ // If a result set generating query was provided, dataset_schema contains the
+ // schema of the dataset as described in Schema.fbs::Schema, it is serialized as an IPC message.
+ bytes dataset_schema = 2;
+
+ // If the query provided contained parameters, parameter_schema contains the
+ // Schema of the expected parameters as described in Schema.fbs::Schema.
+ bytes parameter_schema = 3;
+}
+
+/*
+ * Request message for the "ClosePreparedStatement" action on a
+ * Flight SQL enabled backend.
+ * Closes server resources associated with the prepared statement handle.
+ */
+message ActionClosePreparedStatementRequest {
+ // Opaque handle for the prepared statement on the server.
+ string prepared_statement_handle = 1;
+}
+
+
+// SQL Execution Messages.
+
+/*
+ * Represents a SQL query. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ */
+message CommandStatementQuery {
+ // The SQL syntax.
+ string query = 1;
+}
+
+/*
+ * Represents an instance of executing a prepared statement. Used in the
+ * command member of FlightDescriptor for the following RPC calls:
+ * TODO: Is the idea that a Put with parameter values would execute multiple bound versions of the prepared statement
Review comment:
Thanks. There's been some more progress on the schema evolution proposal so please feel free to chime in there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664138603
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
Review comment:
Slightly revised, but I think still incorrect: `new ArrowType.Union(UnionMode.Dense, new int[] {Type.Utf8, Type.Int})`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r664923300
##########
File path: java/flight/flight-sql/src/main/java/org/apache/arrow/flight/sql/FlightSqlProducer.java
##########
@@ -0,0 +1,541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.flight.sql;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.arrow.flight.Action;
+import org.apache.arrow.flight.ActionType;
+import org.apache.arrow.flight.FlightDescriptor;
+import org.apache.arrow.flight.FlightInfo;
+import org.apache.arrow.flight.FlightProducer;
+import org.apache.arrow.flight.FlightStream;
+import org.apache.arrow.flight.PutResult;
+import org.apache.arrow.flight.Result;
+import org.apache.arrow.flight.SchemaResult;
+import org.apache.arrow.flight.Ticket;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionClosePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSql.ActionCreatePreparedStatementRequest;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetCatalogs;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetSchemas;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetSqlInfo;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetTableTypes;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandGetTables;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementQuery;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandPreparedStatementUpdate;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandStatementQuery;
+import org.apache.arrow.flight.sql.impl.FlightSql.CommandStatementUpdate;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.UnionMode;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.types.pojo.Schema;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+
+import io.grpc.Status;
+
+/**
+ * API to Implement an Arrow Flight SQL producer.
+ */
+public abstract class FlightSqlProducer implements FlightProducer, AutoCloseable {
+ /**
+ * Depending on the provided command, method either:
+ * 1. Return information about a SQL query, or
+ * 2. Return information about a prepared statement. In this case, parameters binding is allowed.
+ *
+ * @param context Per-call context.
+ * @param descriptor The descriptor identifying the data stream.
+ * @return information about the given SQL query, or the given prepared statement.
+ */
+ @Override
+ public FlightInfo getFlightInfo(CallContext context, FlightDescriptor descriptor) {
+ final Any command = FlightSqlUtils.parseOrThrow(descriptor.getCommand());
+
+ if (command.is(CommandStatementQuery.class)) {
+ return getFlightInfoStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandStatementQuery.class), context, descriptor);
+ } else if (command.is(CommandPreparedStatementQuery.class)) {
+ return getFlightInfoPreparedStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandPreparedStatementQuery.class), context, descriptor);
+ } else if (command.is(CommandGetCatalogs.class)) {
+ return getFlightInfoCatalogs(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetCatalogs.class), context, descriptor);
+ } else if (command.is(CommandGetSchemas.class)) {
+ return getFlightInfoSchemas(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetSchemas.class), context, descriptor);
+ } else if (command.is(CommandGetTables.class)) {
+ return getFlightInfoTables(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetTables.class), context, descriptor);
+ } else if (command.is(CommandGetTableTypes.class)) {
+ return getFlightInfoTableTypes(context, descriptor);
+ } else if (command.is(CommandGetSqlInfo.class)) {
+ return getFlightInfoSqlInfo(
+ FlightSqlUtils.unpackOrThrow(command, CommandGetSqlInfo.class), context, descriptor);
+ }
+
+ throw Status.INVALID_ARGUMENT.asRuntimeException();
+ }
+
+ /**
+ * Returns the schema of the result produced by the SQL query.
+ *
+ * @param context Per-call context.
+ * @param descriptor The descriptor identifying the data stream.
+ * @return the result set schema.
+ */
+ @Override
+ public SchemaResult getSchema(CallContext context, FlightDescriptor descriptor) {
+ final Any command = FlightSqlUtils.parseOrThrow(descriptor.getCommand());
+
+ if (command.is(CommandStatementQuery.class)) {
+ return getSchemaStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandStatementQuery.class), context, descriptor);
+ } else if (command.is(CommandGetCatalogs.class)) {
+ return getSchemaCatalogs();
+ } else if (command.is(CommandGetSchemas.class)) {
+ return getSchemaSchemas();
+ } else if (command.is(CommandGetTables.class)) {
+ return getSchemaTables();
+ } else if (command.is(CommandGetTableTypes.class)) {
+ return getSchemaTableTypes();
+ } else if (command.is(CommandGetSqlInfo.class)) {
+ return getSchemaSqlInfo();
+ }
+
+ throw Status.INVALID_ARGUMENT.asRuntimeException();
+ }
+
+ /**
+ * Depending on the provided command, method either:
+ * 1. Return data for a stream produced by executing the provided SQL query, or
+ * 2. Return data for a prepared statement. In this case, parameters binding is allowed.
+ *
+ * @param context Per-call context.
+ * @param ticket The application-defined ticket identifying this stream.
+ * @param listener An interface for sending data back to the client.
+ */
+ @Override
+ public void getStream(CallContext context, Ticket ticket, ServerStreamListener listener) {
+ final Any command;
+
+ try {
+ command = Any.parseFrom(ticket.getBytes());
+ } catch (InvalidProtocolBufferException e) {
+ listener.error(e);
+ return;
+ }
+
+ if (command.is(CommandStatementQuery.class)) {
+ getStreamStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandStatementQuery.class), context, ticket, listener);
+ } else if (command.is(CommandPreparedStatementQuery.class)) {
+ getStreamPreparedStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandPreparedStatementQuery.class), context, ticket, listener);
+ } else if (command.is(CommandGetCatalogs.class)) {
+ getStreamCatalogs(context, ticket, listener);
+ } else if (command.is(CommandGetSchemas.class)) {
+ getStreamSchemas(FlightSqlUtils.unpackOrThrow(command, CommandGetSchemas.class), context, ticket, listener);
+ } else if (command.is(CommandGetTables.class)) {
+ getStreamTables(FlightSqlUtils.unpackOrThrow(command, CommandGetTables.class), context, ticket, listener);
+ } else if (command.is(CommandGetTableTypes.class)) {
+ getStreamTableTypes(context, ticket, listener);
+ } else if (command.is(CommandGetSqlInfo.class)) {
+ getStreamSqlInfo(FlightSqlUtils.unpackOrThrow(command, CommandGetSqlInfo.class), context, ticket, listener);
+ } else {
+ throw Status.INVALID_ARGUMENT.asRuntimeException();
+ }
+ }
+
+ /**
+ * Depending on the provided command, method either:
+ * 1. Execute provided SQL query as an update statement, or
+ * 2. Execute provided update SQL query prepared statement. In this case, parameters binding
+ * is allowed, or
+ * 3. Binds parameters to the provided prepared statement.
+ *
+ * @param context Per-call context.
+ * @param flightStream The data stream being uploaded.
+ * @param ackStream The data stream listener for update result acknowledgement.
+ * @return a Runnable to process the stream.
+ */
+ @Override
+ public Runnable acceptPut(CallContext context, FlightStream flightStream, StreamListener<PutResult> ackStream) {
+ final Any command = FlightSqlUtils.parseOrThrow(flightStream.getDescriptor().getCommand());
+
+ if (command.is(CommandStatementUpdate.class)) {
+ return acceptPutStatement(
+ FlightSqlUtils.unpackOrThrow(command, CommandStatementUpdate.class),
+ context, flightStream, ackStream);
+
+ } else if (command.is(CommandPreparedStatementUpdate.class)) {
+ return acceptPutPreparedStatementUpdate(
+ FlightSqlUtils.unpackOrThrow(command, CommandPreparedStatementUpdate.class),
+ context, flightStream, ackStream);
+
+ } else if (command.is(CommandPreparedStatementQuery.class)) {
+ return acceptPutPreparedStatementQuery(
+ FlightSqlUtils.unpackOrThrow(command, CommandPreparedStatementQuery.class),
+ context, flightStream, ackStream);
+ }
+
+ throw Status.INVALID_ARGUMENT.asRuntimeException();
Review comment:
I looked at the caller, and this is set on the listener as the caller wraps this already. I think it would be redundant to do so again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kylepbit commented on a change in pull request #9368: [WIP] [POC] Flight SQL
Posted by GitBox <gi...@apache.org>.
kylepbit commented on a change in pull request #9368:
URL: https://github.com/apache/arrow/pull/9368#discussion_r661630798
##########
File path: format/FlightSQL.proto
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+option java_package = "org.apache.arrow.flight.sql.impl";
+package arrow.flight.protocol.sql;
+
+/*
+ * Represents a metadata request. Used in the command member of FlightDescriptor
+ * for the following RPC calls:
+ * - GetSchema: return the schema of the query.
+ * - GetFlightInfo: execute the query.
+ *
+ * The returned schema will be:
+ * <
+ * info_name: utf8,
+ * value: dense_union<string_value: string, int_value: int32, bigint_value: int64, int32_bitmask: int32>
+ * >
+ * where there is one row per requested piece of metadata information.
+ */
+message CommandGetSqlInfo {
+ /*
+ * Values are modelled after ODBC's SQLGetInfo() function. This information is intended to provide
+ * Flight SQL clients with basic, SQL syntax and SQL functions related information.
+ * More information types can be added in future releases.
+ * E.g. more SQL syntax support types, scalar functions support, type conversion support etc.
+ *
+ * // TODO: Flesh out the available set of metadata below.
+ *
+ * Initially, Flight SQL will support the following information types:
+ * - Server Information
+ * - Syntax Information
+ *
+ * 1. Server Information: Provides basic information about the Flight SQL Server.
+ *
+ * The name of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_NAME
+ *
+ * The native version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_VERSION
+ *
+ * The Arrow version of the Flight SQL Server.
+ * FLIGHT_SQL_SERVER_ARROW_VERSION
+ *
+ * Indicates whether the Flight SQL Server is read only.
+ * FLIGHT_SQL_SERVER_READ_ONLY
+ *
+ * 2. SQL Syntax Information: provides information about SQL syntax supported by the Flight SQL Server.
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of catalogs.
+ * In a SQL environment, a catalog is a collection of schemas.
+ * SQL_DDL_CATALOG
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of schemas.
+ * In a SQL environment, a catalog is a collection of tables, views, indexes etc.
+ * SQL_DDL_SCHEMA
+ *
+ * Indicates whether the Flight SQL Server supports CREATE and DROP of tables.
+ * In a SQL environment, a table is a collection of rows of information. Each row of information
+ * may have one or more columns of data.
+ * SQL_DDL_TABLE
+ *
+ * Indicates the case sensitivity of catalog, table and schema names.
+ * SQL_IDENTIFIER_CASE
+ *
+ * Indicates the supported character(s) used to surround a delimited identifier.
+ * SQL_IDENTIFIER_QUOTE_CHAR
+ *
+ * Indicates case sensitivity of quoted identifiers.
+ * SQL_QUOTED_IDENTIFIER_CASE
+ *
+ */
+ repeated string info = 1;
Review comment:
I would argue no - this was put in before I started to work on this. My problem is that there's no way to report on what custom_info is present, and since it's custom, this will end up needing specialization.
We can put in a clarification that the list above is the minimum, but sources are free to include other metadata as desired, and then do away with `custom_info`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org