You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "nrg4878 (via GitHub)" <gi...@apache.org> on 2023/09/18 19:37:55 UTC
[GitHub] [hive] nrg4878 opened a new pull request, #4720: HIVE-27597: Implement data connector for Hive to Hive federation over…
nrg4878 opened a new pull request, #4720:
URL: https://github.com/apache/hive/pull/4720
… JDBC (Naveen Gangam)
### What changes were proposed in this pull request?
In HIVE-23291 and HIVE-23302, we added support adding a JDBCStorageHandler based table for remote Hive table. This enables defining a table in hive on one cluster that points to a table in another cluster. Users can perform queries (selects, joins, ctas etc) between this JDBC-storage-handler based table and other local tables.
However, it becomes cumbersome to map each table individually between these clusters. With HIVE-24396, we added support for connectors that allow us to map databases instead of tables. The table metadata is mapped at runtime over a live connector. This allows hive to pull the latest table metadata instead of having a static metadata in HMS.
This PR does not support Kerberos at this point. But that could be a separate enhancement. (the mechanics of kerberos auth may not fit into the current way it works for password-based data sources)
**create connector hive_jdbc
type 'hivejdbc'
url 'jdbc:hive2://nightly.hive.apache.org:10000'
with DCPROPERTIES (
"hive.sql.dbcp.username"="hive",
"hive.sql.dbcp.password"="hive"
);**
**create remote database nightly_default using hive_jdbc with DBPROPERTIES("connector.remoteDbName"="default");**
Sample working demo:
0: jdbc:hive2://localhost:10000> **_create connector hive_jdbc type 'hivejdbc' url 'jdbc:hive2://nightly7x-us-bs-4.nightly7x-us-bs.root.hwx.site:10000' with DCPROPERTIES ("hive.sql.dbcp.username"="hive", "hive.sql.dbcp.password"="hive");_**
INFO : Compiling command(queryId=ngangam_20230918152209_83f300d9-58e1-42fd-a7b4-b5c82e5bbc63): create connector hive_jdbc type 'hivejdbc' url 'jdbc:hive2://nightly7x-us-bs-4.nightly7x-us-bs.root.hwx.site:10000' with DCPROPERTIES ("hive.sql.dbcp.username"="hive", "hive.sql.dbcp.password"="hive")
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling command(queryId=ngangam_20230918152209_83f300d9-58e1-42fd-a7b4-b5c82e5bbc63); Time taken: 1.698 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Executing command(queryId=ngangam_20230918152209_83f300d9-58e1-42fd-a7b4-b5c82e5bbc63): create connector hive_jdbc type 'hivejdbc' url 'jdbc:hive2://nightly7x-us-bs-4.nightly7x-us-bs.root.hwx.site:10000' with DCPROPERTIES ("hive.sql.dbcp.username"="hive", "hive.sql.dbcp.password"="hive")
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=ngangam_20230918152209_83f300d9-58e1-42fd-a7b4-b5c82e5bbc63); Time taken: 0.25 seconds
INFO : OK
No rows affected (2.274 seconds)
0: jdbc:hive2://localhost:10000> **_create remote database nightly_default using hive_jdbc with DBPROPERTIES("connector.remoteDbName"="default");_**
INFO : Compiling command(queryId=ngangam_20230918152225_58e8bfd4-9cf0-4505-9aa3-012be57e23d6): create remote database nightly_default using hive_jdbc with DBPROPERTIES("connector.remoteDbName"="default")
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling command(queryId=ngangam_20230918152225_58e8bfd4-9cf0-4505-9aa3-012be57e23d6); Time taken: 0.014 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Executing command(queryId=ngangam_20230918152225_58e8bfd4-9cf0-4505-9aa3-012be57e23d6): create remote database nightly_default using hive_jdbc with DBPROPERTIES("connector.remoteDbName"="default")
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=ngangam_20230918152225_58e8bfd4-9cf0-4505-9aa3-012be57e23d6); Time taken: 0.06 seconds
INFO : OK
No rows affected (0.092 seconds)
0: jdbc:hive2://localhost:10000> **_use nightly_default;_**
INFO : Compiling command(queryId=ngangam_20230918152528_7af75a5a-54a3-48f0-90cb-209cf236c204): use nightly_default
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling command(queryId=ngangam_20230918152528_7af75a5a-54a3-48f0-90cb-209cf236c204); Time taken: 0.003 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Executing command(queryId=ngangam_20230918152528_7af75a5a-54a3-48f0-90cb-209cf236c204): use nightly_default
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=ngangam_20230918152528_7af75a5a-54a3-48f0-90cb-209cf236c204); Time taken: 0.158 seconds
INFO : OK
No rows affected (0.176 seconds)
0: jdbc:hive2://localhost:10000> **_show tables;_**
INFO : Compiling command(queryId=ngangam_20230918152530_5b5c754b-db5f-47ae-a880-b181757cfa4e): show tables
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling command(queryId=ngangam_20230918152530_5b5c754b-db5f-47ae-a880-b181757cfa4e); Time taken: 0.01 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Executing command(queryId=ngangam_20230918152530_5b5c754b-db5f-47ae-a880-b181757cfa4e): show tables
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=ngangam_20230918152530_5b5c754b-db5f-47ae-a880-b181757cfa4e); Time taken: 3.8 seconds
INFO : OK
+------------+
| tab_name |
+------------+
| sample_07 |
| sample_08 |
| web_logs |
+------------+
3 rows selected (3.84 seconds)
0: jdbc:hive2://localhost:10000> **_describe formatted web_logs;_**
INFO : Compiling command(queryId=ngangam_20230918152542_170176a0-92ff-408b-9133-eab9a1953c23): describe formatted web_logs
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:col_name, type:string, comment:from deserializer), FieldSchema(name:data_type, type:string, comment:from deserializer), FieldSchema(name:comment, type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling command(queryId=ngangam_20230918152542_170176a0-92ff-408b-9133-eab9a1953c23); Time taken: 0.661 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Executing command(queryId=ngangam_20230918152542_170176a0-92ff-408b-9133-eab9a1953c23): describe formatted web_logs
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=ngangam_20230918152542_170176a0-92ff-408b-9133-eab9a1953c23); Time taken: 2.704 seconds
INFO : OK
+-------------------------------+------------------------------------------------+----------------------------------------------------+
| col_name | data_type | comment |
+-------------------------------+------------------------------------------------+----------------------------------------------------+
| _version_ | bigint | from deserializer |
| app | string | from deserializer |
| bytes | int | from deserializer |
| city | string | from deserializer |
| client_ip | string | from deserializer |
| code | smallint | from deserializer |
| country_code | string | from deserializer |
| country_code3 | string | from deserializer |
| country_name | string | from deserializer |
| device_family | string | from deserializer |
| extension | string | from deserializer |
| latitude | float | from deserializer |
| longitude | float | from deserializer |
| method | string | from deserializer |
| os_family | string | from deserializer |
| os_major | string | from deserializer |
| protocol | string | from deserializer |
| record | string | from deserializer |
| referer | string | from deserializer |
| region_code | string | from deserializer |
| request | string | from deserializer |
| subapp | string | from deserializer |
| time | string | from deserializer |
| url | string | from deserializer |
| user_agent | string | from deserializer |
| user_agent_family | string | from deserializer |
| user_agent_major | string | from deserializer |
| id | string | from deserializer |
| date | string | from deserializer |
| | NULL | NULL |
| # Detailed Table Information | NULL | NULL |
| Database: | nightly_default | NULL |
| OwnerType: | USER | NULL |
| Owner: | null | NULL |
| CreateTime: | UNKNOWN | NULL |
| LastAccessTime: | UNKNOWN | NULL |
| Retention: | 0 | NULL |
| Location: | file:/tmp/warehouse/external/web_logs | NULL |
| Table Type: | EXTERNAL_TABLE | NULL |
| Table Parameters: | NULL | NULL |
| | EXTERNAL | TRUE |
| | hive.sql.database.type | HIVE |
| | hive.sql.dbcp.password | hive |
| | hive.sql.dbcp.username | hive |
| | hive.sql.jdbc.driver | org.apache.hive.jdbc.HiveDriver |
| | hive.sql.jdbc.url | jdbc:hive2://nightly7x-us-bs-4.nightly7x-us-bs.root.hwx.site:10000 |
| | hive.sql.schema | default |
| | hive.sql.table | web_logs |
| | storage_handler | org.apache.hive.storage.jdbc.JdbcStorageHandler |
| | NULL | NULL |
| # Storage Information | NULL | NULL |
| SerDe Library: | org.apache.hive.storage.jdbc.JdbcSerDe | NULL |
| InputFormat: | org.apache.hive.storage.jdbc.JdbcInputFormat | NULL |
| OutputFormat: | org.apache.hive.storage.jdbc.JdbcOutputFormat | NULL |
| Compressed: | No | NULL |
| Num Buckets: | 0 | NULL |
| Bucket Columns: | [] | NULL |
| Sort Columns: | [] | NULL |
| Storage Desc Params: | NULL | NULL |
| | serialization.format | 1 |
+-------------------------------+------------------------------------------------+----------------------------------------------------+
60 rows selected (3.392 seconds)
0: jdbc:hive2://localhost:10000> **_describe formatted sample_07;_**
INFO : Compiling command(queryId=ngangam_20230918152648_6ea35ff8-324c-427a-96a5-9007835202f6): describe formatted sample_07
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:col_name, type:string, comment:from deserializer), FieldSchema(name:data_type, type:string, comment:from deserializer), FieldSchema(name:comment, type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling command(queryId=ngangam_20230918152648_6ea35ff8-324c-427a-96a5-9007835202f6); Time taken: 0.641 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Executing command(queryId=ngangam_20230918152648_6ea35ff8-324c-427a-96a5-9007835202f6): describe formatted sample_07
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=ngangam_20230918152648_6ea35ff8-324c-427a-96a5-9007835202f6); Time taken: 2.532 seconds
INFO : OK
+-------------------------------+------------------------------------------------+----------------------------------------------------+
| col_name | data_type | comment |
+-------------------------------+------------------------------------------------+----------------------------------------------------+
| code | string | from deserializer |
| description | string | from deserializer |
| total_emp | int | from deserializer |
| salary | int | from deserializer |
| | NULL | NULL |
| # Detailed Table Information | NULL | NULL |
| Database: | nightly_default | NULL |
| OwnerType: | USER | NULL |
| Owner: | null | NULL |
| CreateTime: | UNKNOWN | NULL |
| LastAccessTime: | UNKNOWN | NULL |
| Retention: | 0 | NULL |
| Location: | file:/tmp/warehouse/external/sample_07 | NULL |
| Table Type: | EXTERNAL_TABLE | NULL |
| Table Parameters: | NULL | NULL |
| | EXTERNAL | TRUE |
| | hive.sql.database.type | HIVE |
| | hive.sql.dbcp.password | hive |
| | hive.sql.dbcp.username | hive |
| | hive.sql.jdbc.driver | org.apache.hive.jdbc.HiveDriver |
| | hive.sql.jdbc.url | jdbc:hive2://nightly7x-us-bs-4.nightly7x-us-bs.root.hwx.site:10000 |
| | hive.sql.schema | default |
| | hive.sql.table | sample_07 |
| | storage_handler | org.apache.hive.storage.jdbc.JdbcStorageHandler |
| | NULL | NULL |
| # Storage Information | NULL | NULL |
| SerDe Library: | org.apache.hive.storage.jdbc.JdbcSerDe | NULL |
| InputFormat: | org.apache.hive.storage.jdbc.JdbcInputFormat | NULL |
| OutputFormat: | org.apache.hive.storage.jdbc.JdbcOutputFormat | NULL |
| Compressed: | No | NULL |
| Num Buckets: | 0 | NULL |
| Bucket Columns: | [] | NULL |
| Sort Columns: | [] | NULL |
| Storage Desc Params: | NULL | NULL |
| | serialization.format | 1 |
+-------------------------------+------------------------------------------------+----------------------------------------------------+
35 rows selected (3.198 seconds)
0: jdbc:hive2://localhost:10000> **_set hive.execution.engine=mr;_**
Error: Error while processing statement: hive execution engine mr is not supported. (state=42000,code=1)
0: jdbc:hive2://localhost:10000> **_select count(*) from sample_07 where salary > 50000;_**
INFO : Compiling command(queryId=ngangam_20230918152746_0cce0ed6-58ab-477a-805b-a97dca077ff1): select count(*) from sample_07 where salary > 50000
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null)
INFO : Completed compiling command(queryId=ngangam_20230918152746_0cce0ed6-58ab-477a-805b-a97dca077ff1); Time taken: 5.674 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Executing command(queryId=ngangam_20230918152746_0cce0ed6-58ab-477a-805b-a97dca077ff1): select count(*) from sample_07 where salary > 50000
WARN : Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, impala) or using Hive 1.X releases.
INFO : Query ID = ngangam_20230918152746_0cce0ed6-58ab-477a-805b-a97dca077ff1
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Number of reduce tasks determined at compile time: 1
INFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=<number>
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=<number>
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=<number>
INFO : number of splits:1
INFO : Submitting tokens for job: job_local1498163518_0001
INFO : Executing with tokens: []
INFO : The url to track the job: http://localhost:8080/
INFO : Job running in-process (local Hadoop)
INFO : 2023-09-18 15:27:55,391 Stage-1 map = 0%, reduce = 0%
INFO : 2023-09-18 15:27:59,418 Stage-1 map = 100%, reduce = 100%
INFO : Ended Job = job_local1498163518_0001
INFO : MapReduce Jobs Launched:
INFO : Stage-Stage-1: HDFS Read: 0 HDFS Write: 0 SUCCESS
INFO : Total MapReduce CPU Time Spent: 0 msec
INFO : Completed executing command(queryId=ngangam_20230918152746_0cce0ed6-58ab-477a-805b-a97dca077ff1); Time taken: 7.418 seconds
INFO : OK
+------+
| _c0 |
+------+
| 292 |
+------+
1 row selected (13.157 seconds)
0: jdbc:hive2://localhost:10000>
### Why are the changes needed?
This connector enables users to map databases between 2 hive clusters and federate queries over JDBC between tables in the 2 clusters.
### Does this PR introduce _any_ user-facing change?
NO
### Is the change a dependency upgrade?
NO
### How was this patch tested?
Manually
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zhangbutao commented on a diff in pull request #4720: HIVE-27597: Implement data connector for Hive to Hive federation over…
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #4720:
URL: https://github.com/apache/hive/pull/4720#discussion_r1329946152
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(scoped_db, null, null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + sqle);
+ }
+ return rs;
+ }
+
+ /**
+ * Fetch a single table with the given name, returns a Hive Table object from the remote database
+ * @return Table A Table object for the matching table, null otherwise.
+ * @throws MetaException To indicate any failures with executing this API
+ * @param tableName
+ */
+ @Override public ResultSet fetchTableMetadata(String tableName) throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getColumns(null, scoped_db, tableName, null);
+ } catch (Exception ex) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + ex.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + ex);
+ }
+ return rs;
+}
+
+ @Override protected String getCatalogName() {
+ return null;
+ }
+
+ @Override protected String getDatabaseName() {
+ return scoped_db;
Review Comment:
A interesting case: if this is a `remotedb` type db. Can we create a remotedb against this db?
I have tested this case, and it didn't seem ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn commented on a diff in pull request #4720: HIVE-27597: Implement data connector for Hive to Hive federation over…
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #4720:
URL: https://github.com/apache/hive/pull/4720#discussion_r1336724912
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
Review Comment:
I would have gone with adding the methods, but no hard feelings if you want to go ahead with another approach
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn commented on a diff in pull request #4720: HIVE-27597: Implement data connector for Hive to Hive federation over…
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #4720:
URL: https://github.com/apache/hive/pull/4720#discussion_r1329594335
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
Review Comment:
this can be final
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
Review Comment:
I don't think intern is required for compile time constant?
https://stackoverflow.com/questions/23720381/when-is-interning-during-done-during-compile-time-or-run-time-the-reason-for-su
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(scoped_db, null, null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + sqle);
+ }
+ return rs;
+ }
+
+ /**
+ * Fetch a single table with the given name, returns a Hive Table object from the remote database
+ * @return Table A Table object for the matching table, null otherwise.
+ * @throws MetaException To indicate any failures with executing this API
+ * @param tableName
+ */
+ @Override public ResultSet fetchTableMetadata(String tableName) throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getColumns(null, scoped_db, tableName, null);
+ } catch (Exception ex) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + ex.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + ex);
+ }
+ return rs;
+}
+
+ @Override protected String getCatalogName() {
+ return null;
+ }
+
+ @Override protected String getDatabaseName() {
+ return scoped_db;
+ }
+
+ protected String getDataType(String dbDataType, int size) {
+ String mappedType = super.getDataType(dbDataType, size);
+ if (!mappedType.equalsIgnoreCase(ColumnType.VOID_TYPE_NAME)) {
+ return mappedType;
+ }
+
+ // map any db specific types here.
+ switch (dbDataType.toLowerCase())
+ {
+ case "string":
+ mappedType = ColumnType.STRING_TYPE_NAME;
+ break;
+ default:
+ mappedType = ColumnType.VOID_TYPE_NAME;
+ break;
+ }
Review Comment:
there are only 2 cases? we could have used if-else
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(scoped_db, null, null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + sqle);
+ }
+ return rs;
+ }
+
+ /**
+ * Fetch a single table with the given name, returns a Hive Table object from the remote database
+ * @return Table A Table object for the matching table, null otherwise.
+ * @throws MetaException To indicate any failures with executing this API
+ * @param tableName
+ */
+ @Override public ResultSet fetchTableMetadata(String tableName) throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getColumns(null, scoped_db, tableName, null);
+ } catch (Exception ex) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + ex.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + ex);
+ }
+ return rs;
+}
+
+ @Override protected String getCatalogName() {
+ return null;
+ }
+
+ @Override protected String getDatabaseName() {
+ return scoped_db;
+ }
+
+ protected String getDataType(String dbDataType, int size) {
Review Comment:
missing ``Override`` annotation
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(scoped_db, null, null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + sqle);
+ }
+ return rs;
+ }
+
+ /**
+ * Fetch a single table with the given name, returns a Hive Table object from the remote database
+ * @return Table A Table object for the matching table, null otherwise.
+ * @throws MetaException To indicate any failures with executing this API
+ * @param tableName
+ */
+ @Override public ResultSet fetchTableMetadata(String tableName) throws MetaException {
Review Comment:
this method is also dupe in 3 places:
MSSQLConnectorProvider, OracleConnectorProvider & here. maybe now or later we should consider a refactor into a common place and use
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(scoped_db, null, null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + sqle);
+ }
+ return rs;
+ }
+
+ /**
+ * Fetch a single table with the given name, returns a Hive Table object from the remote database
+ * @return Table A Table object for the matching table, null otherwise.
+ * @throws MetaException To indicate any failures with executing this API
+ * @param tableName
+ */
+ @Override public ResultSet fetchTableMetadata(String tableName) throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getColumns(null, scoped_db, tableName, null);
+ } catch (Exception ex) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + ex.getMessage());
Review Comment:
should use logger format
```
LOG.warn("Could not retrieve table names from remote datasource, cause: {}", ex.getMessage());
```
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
Review Comment:
this is dupe in 3 classes:
here, DerbySQLConnectorProvider.java, MySQLConnectorProvider.java
may be we should consider a refactor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27597: Implement data connector for Hive to Hive federation over… [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4720:
URL: https://github.com/apache/hive/pull/4720#issuecomment-1751797809
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4720)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4720&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4720&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4720&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=CODE_SMELL) [28 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4720&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4720&metric=duplicated_lines_density&view=list) No Duplication information
 The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4720: HIVE-27597: Implement data connector for Hive to Hive federation over…
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4720:
URL: https://github.com/apache/hive/pull/4720#issuecomment-1724344108
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4720)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4720&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4720&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4720&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4720&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4720&metric=duplicated_lines_density&view=list) No Duplication information
 The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27597: Implement data connector for Hive to Hive federation over… [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #4720:
URL: https://github.com/apache/hive/pull/4720#issuecomment-1751936767
> I understand the need for hive-jdbc jar (and I assumed it was already on the Tez CP) but do you know what class it needed from the hive-service.jar? We should definitely document this? Would you mind adding a wiki to the hive docs ( we need to document this feature anyhow)
@nrg4878 If you submit a query which joins two hive-jdbc remote table, a tez task should be launched. And if hive-service.jar is not in Tez classpath, Tez task will faill with the following exception:
```
java.lang.NoClassDefFoundError: org/apache/hive/service/cli/HiveSQLException
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:297)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:285)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:94)
at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52)
at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:374)
at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:106)
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:649)
at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:531)
at org.apache.commons.dbcp2.BasicDataSource.getLogWriter(BasicDataSource.java:1043)
at org.apache.commons.dbcp2.BasicDataSourceFactory.createDataSource(BasicDataSourceFactory.java:461)
at org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.initializeDatabaseConnection(GenericJdbcDatabaseAccessor.java:425)
at org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getRecordIterator(GenericJdbcDatabaseAccessor.java:231)
at org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:58)
```
I'd love to add this note to the hive docs, and i think we maybe document here?https://cwiki.apache.org/confluence/display/Hive/Data+Connectors+in+Hive
Could you please give me write access on this docs page? My usename is `zhangbutao`, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27597: Implement data connector for Hive to Hive federation over… [hive]
Posted by "nrg4878 (via GitHub)" <gi...@apache.org>.
nrg4878 commented on PR #4720:
URL: https://github.com/apache/hive/pull/4720#issuecomment-1751288260
@zhangbutao Thank for testing this and providing additional set of eyes.
> I have tested this change manually in my local hive cluster. This change is mostly ok for me. I have some considerations and maybe we can improve this feature in the future.
>
> 1. Support kerberos. Most production clusters will enable kerberos, and we need to think how to support this?
Agreed. This is a must-do. I started to look into this a bit and I stumbled into roadblock on how to do this when each connector would have its own keytab file. I wasnt sure how to setup the /etc/krb5.conf file. This is going to need some cycles.
> 2. Can we guarantee `Driver ` & `JDBC `compatibility between different Hive version. IMO, `hive-exec.jar` and `hive-jdbc.jar` are the mainly jar which are used by jdbc connection. And if I want to use Hive4 to query another Hive3 or Hive2, can we success?
This should work. The above output was from a testing the connector deployed on apache-master (4.x) connecting to a HS2 cluster running hive-3.1+. While this proves this works, you have a good point. We need to ensure that they remain compatible. We haven't changed the thrift IDL for HS2 in a very long time.
> BTW, in my local cluster, i need to add `hive-jdbc-4.0.0-beta-2-SNAPSHOT.jar` & `hive-service-4.0.0-beta-2-SNAPSHOT.jar` in Tez&MR AppMaster classpath, otherwise, it will fail in case of launching a Tez&MR task. This is not a problem, It's just a reminder for other folks who want to try this feature.
I understand the need for hive-jdbc jar (and I assumed it was already on the Tez CP) but do you know what class it needed from the hive-service.jar? We should definitely document this? Would you mind adding a wiki to the hive docs ( we need to document this feature anyhow)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27597: Implement data connector for Hive to Hive federation over… [hive]
Posted by "nrg4878 (via GitHub)" <gi...@apache.org>.
nrg4878 merged PR #4720:
URL: https://github.com/apache/hive/pull/4720
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27597: Implement data connector for Hive to Hive federation over… [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4720:
URL: https://github.com/apache/hive/pull/4720#issuecomment-1751850819
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4720)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4720&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4720&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4720&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=CODE_SMELL) [28 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4720&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4720&metric=duplicated_lines_density&view=list) No Duplication information
 The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27597: Implement data connector for Hive to Hive federation over… [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4720:
URL: https://github.com/apache/hive/pull/4720#issuecomment-1751271044
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4720)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4720&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4720&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4720&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=CODE_SMELL) [28 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4720&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4720&metric=duplicated_lines_density&view=list) No Duplication information
 The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zhangbutao commented on a diff in pull request #4720: HIVE-27597: Implement data connector for Hive to Hive federation over…
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #4720:
URL: https://github.com/apache/hive/pull/4720#discussion_r1329946152
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(scoped_db, null, null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + sqle);
+ }
+ return rs;
+ }
+
+ /**
+ * Fetch a single table with the given name, returns a Hive Table object from the remote database
+ * @return Table A Table object for the matching table, null otherwise.
+ * @throws MetaException To indicate any failures with executing this API
+ * @param tableName
+ */
+ @Override public ResultSet fetchTableMetadata(String tableName) throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getColumns(null, scoped_db, tableName, null);
+ } catch (Exception ex) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + ex.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + ex);
+ }
+ return rs;
+}
+
+ @Override protected String getCatalogName() {
+ return null;
+ }
+
+ @Override protected String getDatabaseName() {
+ return scoped_db;
Review Comment:
A intresting case: if this is a `remotedb` type db. Can we create a remotedb against this db?
I have tested this case, and it didn't seem ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] nrg4878 commented on a diff in pull request #4720: HIVE-27597: Implement data connector for Hive to Hive federation over…
Posted by "nrg4878 (via GitHub)" <gi...@apache.org>.
nrg4878 commented on code in PR #4720:
URL: https://github.com/apache/hive/pull/4720#discussion_r1330464405
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(scoped_db, null, null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + sqle);
+ }
+ return rs;
+ }
+
+ /**
+ * Fetch a single table with the given name, returns a Hive Table object from the remote database
+ * @return Table A Table object for the matching table, null otherwise.
+ * @throws MetaException To indicate any failures with executing this API
+ * @param tableName
+ */
+ @Override public ResultSet fetchTableMetadata(String tableName) throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getColumns(null, scoped_db, tableName, null);
+ } catch (Exception ex) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + ex.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + ex);
+ }
+ return rs;
+}
+
+ @Override protected String getCatalogName() {
+ return null;
+ }
+
+ @Override protected String getDatabaseName() {
+ return scoped_db;
+ }
+
+ protected String getDataType(String dbDataType, int size) {
+ String mappedType = super.getDataType(dbDataType, size);
+ if (!mappedType.equalsIgnoreCase(ColumnType.VOID_TYPE_NAME)) {
+ return mappedType;
+ }
+
+ // map any db specific types here.
+ switch (dbDataType.toLowerCase())
+ {
+ case "string":
+ mappedType = ColumnType.STRING_TYPE_NAME;
+ break;
+ default:
+ mappedType = ColumnType.VOID_TYPE_NAME;
+ break;
+ }
Review Comment:
Actually, this reminds of some pending work that I needed to do for complex datatypes. Thanks for the reminder. We will have more cases here.
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
Review Comment:
fixed
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
Review Comment:
we can use 2 new variables for catalog/schema and schema/db and use those or introduce 2 addional methods that subclasses have to implement getSchema(), getDatabase().
How did you want me to refactor?
IMHO, this is the reason we use abstraction so subclasses can provide their implementation.
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(scoped_db, null, null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + sqle);
+ }
+ return rs;
+ }
+
+ /**
+ * Fetch a single table with the given name, returns a Hive Table object from the remote database
+ * @return Table A Table object for the matching table, null otherwise.
+ * @throws MetaException To indicate any failures with executing this API
+ * @param tableName
+ */
+ @Override public ResultSet fetchTableMetadata(String tableName) throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getColumns(null, scoped_db, tableName, null);
+ } catch (Exception ex) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + ex.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + ex);
+ }
+ return rs;
+}
+
+ @Override protected String getCatalogName() {
+ return null;
+ }
+
+ @Override protected String getDatabaseName() {
+ return scoped_db;
+ }
+
+ protected String getDataType(String dbDataType, int size) {
+ String mappedType = super.getDataType(dbDataType, size);
+ if (!mappedType.equalsIgnoreCase(ColumnType.VOID_TYPE_NAME)) {
+ return mappedType;
+ }
+
+ // map any db specific types here.
+ switch (dbDataType.toLowerCase())
+ {
+ case "string":
+ mappedType = ColumnType.STRING_TYPE_NAME;
+ break;
+ default:
+ mappedType = ColumnType.VOID_TYPE_NAME;
+ break;
+ }
+ return mappedType;
+ }
+
+ @Override protected String getDatasourceType() { return "HIVE"; }
Review Comment:
ACK
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(scoped_db, null, null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + sqle);
+ }
+ return rs;
+ }
+
+ /**
+ * Fetch a single table with the given name, returns a Hive Table object from the remote database
+ * @return Table A Table object for the matching table, null otherwise.
+ * @throws MetaException To indicate any failures with executing this API
+ * @param tableName
+ */
+ @Override public ResultSet fetchTableMetadata(String tableName) throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getColumns(null, scoped_db, tableName, null);
+ } catch (Exception ex) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + ex.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + ex);
+ }
+ return rs;
+}
+
+ @Override protected String getCatalogName() {
+ return null;
+ }
+
+ @Override protected String getDatabaseName() {
+ return scoped_db;
Review Comment:
not sure if I understood correctly but I think you are asking if the database we are mapping is already a "mapped/remote" database on the remote HS2 right? I believe the GetTablesOperation implementation uses a different API that looks for tables registered in HMS for this DB. In this case, it would be empty.
Although the idea is neat, I wasnt sure if this would be a good idea. This will need additional work because we have to retain the original set of properties returned without overwriting them (like the URL, type, credentials etc). Not sure what other problems this could pose other than having cyclical references. A simple workaround could be to map that end database directly using a local connector.
Hope this makes sense.
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
Review Comment:
old habit. let me read thru
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(scoped_db, null, null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + sqle);
+ }
+ return rs;
+ }
+
+ /**
+ * Fetch a single table with the given name, returns a Hive Table object from the remote database
+ * @return Table A Table object for the matching table, null otherwise.
+ * @throws MetaException To indicate any failures with executing this API
+ * @param tableName
+ */
+ @Override public ResultSet fetchTableMetadata(String tableName) throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getColumns(null, scoped_db, tableName, null);
+ } catch (Exception ex) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + ex.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + ex);
+ }
+ return rs;
+}
+
+ @Override protected String getCatalogName() {
+ return null;
+ }
+
+ @Override protected String getDatabaseName() {
+ return scoped_db;
+ }
+
+ protected String getDataType(String dbDataType, int size) {
Review Comment:
Missed during porting from CDH code.
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(scoped_db, null, null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + sqle);
+ }
+ return rs;
+ }
+
+ /**
+ * Fetch a single table with the given name, returns a Hive Table object from the remote database
+ * @return Table A Table object for the matching table, null otherwise.
+ * @throws MetaException To indicate any failures with executing this API
+ * @param tableName
+ */
+ @Override public ResultSet fetchTableMetadata(String tableName) throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getColumns(null, scoped_db, tableName, null);
+ } catch (Exception ex) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + ex.getMessage());
Review Comment:
ACK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zhangbutao commented on pull request #4720: HIVE-27597: Implement data connector for Hive to Hive federation over…
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #4720:
URL: https://github.com/apache/hive/pull/4720#issuecomment-1725129889
I have tested this change manually in my local hive cluster. This change is mostly ok for me. I have some considerations and maybe we can improve this feature in the future.
1. Support kerberos. Most production clusters will enable kerberos, and we need to think how to support this?
2. Support Hive_A to query another Hive_B which is on different federated HDFS cluster. Because the current Hive_A can not get the another federated HDFS conf, so it will thow `UnknownHostException` when querying data. Maybe we need to think how to add the another hdfs configuration(hdfs-site.xml & core-site.xml) in the current Hive?
3. Can we guarantee `Driver ` & `JDBC `compatibility between different Hive version. IMO, `hive-exec.jar` and `hive-jdbc.jar` are the mainly jar which are used by jdbc connection. And if I want to use Hive4 to query another Hive3 or Hive2, can we success?
BTW, in my local cluster, i need to add `hive-jdbc-4.0.0-beta-2-SNAPSHOT.jar` & `hive-service-4.0.0-beta-2-SNAPSHOT.jar` in Tez&MR AppMaster classpath, otherwise, it will fail in case of launching a Tez&MR task. This is not a problem, It's just a reminder for other folks who want to try this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27597: Implement data connector for Hive to Hive federation over… [hive]
Posted by "nrg4878 (via GitHub)" <gi...@apache.org>.
nrg4878 commented on code in PR #4720:
URL: https://github.com/apache/hive/pull/4720#discussion_r1349142912
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/AbstractJDBCConnectorProvider.java:
##########
@@ -234,9 +234,32 @@ protected boolean isClosed() {
return null;
}
- protected abstract ResultSet fetchTableMetadata(String tableName) throws MetaException;
+ protected ResultSet fetchTableMetadata(String tableName) throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(getCatalogName(), getDatabaseName(), null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ }
+ return rs;
+ }
- protected abstract ResultSet fetchTableNames() throws MetaException;
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(getCatalogName(), getDatabaseName(), null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
Review Comment:
fixed
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/AbstractJDBCConnectorProvider.java:
##########
@@ -415,4 +438,7 @@ protected Properties getConnectionProperties() {
connectorPropMap.forEach((k, v) -> connectionProperties.put(k, v));
return connectionProperties;
}
-}
\ No newline at end of file
+
+ @Override
+ protected String getDatasourceType() { return type; };
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27597: Implement data connector for Hive to Hive federation over… [hive]
Posted by "nrg4878 (via GitHub)" <gi...@apache.org>.
nrg4878 commented on code in PR #4720:
URL: https://github.com/apache/hive/pull/4720#discussion_r1348233253
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(scoped_db, null, null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + sqle);
+ }
+ return rs;
+ }
+
+ /**
+ * Fetch a single table with the given name, returns a Hive Table object from the remote database
+ * @return Table A Table object for the matching table, null otherwise.
+ * @throws MetaException To indicate any failures with executing this API
+ * @param tableName
+ */
+ @Override public ResultSet fetchTableMetadata(String tableName) throws MetaException {
Review Comment:
fixed
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27597: Implement data connector for Hive to Hive federation over… [hive]
Posted by "nrg4878 (via GitHub)" <gi...@apache.org>.
nrg4878 commented on PR #4720:
URL: https://github.com/apache/hive/pull/4720#issuecomment-1753227942
Thank you for the review @ayushtkn and @zhangbutao.
@zhangbutao The Dataconnectors doc page would be a perfect place. I am not sure how to grant you edit access. But I will give it a try
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27597: Implement data connector for Hive to Hive federation over… [hive]
Posted by "nrg4878 (via GitHub)" <gi...@apache.org>.
nrg4878 commented on code in PR #4720:
URL: https://github.com/apache/hive/pull/4720#discussion_r1348234602
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(scoped_db, null, null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + sqle);
+ }
+ return rs;
+ }
+
+ /**
+ * Fetch a single table with the given name, returns a Hive Table object from the remote database
+ * @return Table A Table object for the matching table, null otherwise.
+ * @throws MetaException To indicate any failures with executing this API
+ * @param tableName
+ */
+ @Override public ResultSet fetchTableMetadata(String tableName) throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getColumns(null, scoped_db, tableName, null);
+ } catch (Exception ex) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + ex.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + ex);
+ }
+ return rs;
+}
+
+ @Override protected String getCatalogName() {
+ return null;
+ }
+
+ @Override protected String getDatabaseName() {
+ return scoped_db;
+ }
+
+ protected String getDataType(String dbDataType, int size) {
+ String mappedType = super.getDataType(dbDataType, size);
+ if (!mappedType.equalsIgnoreCase(ColumnType.VOID_TYPE_NAME)) {
+ return mappedType;
+ }
+
+ // map any db specific types here.
+ switch (dbDataType.toLowerCase())
+ {
+ case "string":
+ mappedType = ColumnType.STRING_TYPE_NAME;
+ break;
+ default:
+ mappedType = ColumnType.VOID_TYPE_NAME;
+ break;
+ }
Review Comment:
I tried to add support for complex types that hive supports.
JdbcSerDe does not support it currently
https://github.com/apache/hive/blob/master/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcSerDe.java#L126
I have tried to add support for it here, but this is going to take a bit more time. We can add this support in a follow on jira.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27597: Implement data connector for Hive to Hive federation over… [hive]
Posted by "nrg4878 (via GitHub)" <gi...@apache.org>.
nrg4878 commented on code in PR #4720:
URL: https://github.com/apache/hive/pull/4720#discussion_r1348234695
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(scoped_db, null, null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + sqle);
+ }
+ return rs;
+ }
+
+ /**
+ * Fetch a single table with the given name, returns a Hive Table object from the remote database
+ * @return Table A Table object for the matching table, null otherwise.
+ * @throws MetaException To indicate any failures with executing this API
+ * @param tableName
+ */
+ @Override public ResultSet fetchTableMetadata(String tableName) throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getColumns(null, scoped_db, tableName, null);
+ } catch (Exception ex) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + ex.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + ex);
+ }
+ return rs;
+}
+
+ @Override protected String getCatalogName() {
+ return null;
+ }
+
+ @Override protected String getDatabaseName() {
+ return scoped_db;
+ }
+
+ protected String getDataType(String dbDataType, int size) {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27597: Implement data connector for Hive to Hive federation over… [hive]
Posted by "nrg4878 (via GitHub)" <gi...@apache.org>.
nrg4878 commented on code in PR #4720:
URL: https://github.com/apache/hive/pull/4720#discussion_r1349143665
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/AbstractJDBCConnectorProvider.java:
##########
@@ -234,9 +234,32 @@ protected boolean isClosed() {
return null;
}
- protected abstract ResultSet fetchTableMetadata(String tableName) throws MetaException;
+ protected ResultSet fetchTableMetadata(String tableName) throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(getCatalogName(), getDatabaseName(), null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zhangbutao commented on a diff in pull request #4720: HIVE-27597: Implement data connector for Hive to Hive federation over…
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #4720:
URL: https://github.com/apache/hive/pull/4720#discussion_r1329779317
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(scoped_db, null, null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + sqle);
+ }
+ return rs;
+ }
+
+ /**
+ * Fetch a single table with the given name, returns a Hive Table object from the remote database
+ * @return Table A Table object for the matching table, null otherwise.
+ * @throws MetaException To indicate any failures with executing this API
+ * @param tableName
+ */
+ @Override public ResultSet fetchTableMetadata(String tableName) throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getColumns(null, scoped_db, tableName, null);
+ } catch (Exception ex) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + ex.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + ex);
+ }
+ return rs;
+}
+
+ @Override protected String getCatalogName() {
+ return null;
+ }
+
+ @Override protected String getDatabaseName() {
+ return scoped_db;
+ }
+
+ protected String getDataType(String dbDataType, int size) {
+ String mappedType = super.getDataType(dbDataType, size);
+ if (!mappedType.equalsIgnoreCase(ColumnType.VOID_TYPE_NAME)) {
+ return mappedType;
+ }
+
+ // map any db specific types here.
+ switch (dbDataType.toLowerCase())
+ {
+ case "string":
+ mappedType = ColumnType.STRING_TYPE_NAME;
+ break;
+ default:
+ mappedType = ColumnType.VOID_TYPE_NAME;
+ break;
+ }
+ return mappedType;
+ }
+
+ @Override protected String getDatasourceType() { return "HIVE"; }
Review Comment:
nit: Can return a constant or enum.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] nrg4878 commented on pull request #4720: HIVE-27597: Implement data connector for Hive to Hive federation over…
Posted by "nrg4878 (via GitHub)" <gi...@apache.org>.
nrg4878 commented on PR #4720:
URL: https://github.com/apache/hive/pull/4720#issuecomment-1724257493
@zhangbutao Can you please review this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zhangbutao commented on a diff in pull request #4720: HIVE-27597: Implement data connector for Hive to Hive federation over…
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #4720:
URL: https://github.com/apache/hive/pull/4720#discussion_r1332337891
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
+
+ public HiveJDBCConnectorProvider(String dbName, DataConnector dataConn) {
+ super(dbName, dataConn, DRIVER_CLASS);
+ }
+
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ @Override protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(scoped_db, null, null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + sqle);
+ }
+ return rs;
+ }
+
+ /**
+ * Fetch a single table with the given name, returns a Hive Table object from the remote database
+ * @return Table A Table object for the matching table, null otherwise.
+ * @throws MetaException To indicate any failures with executing this API
+ * @param tableName
+ */
+ @Override public ResultSet fetchTableMetadata(String tableName) throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getColumns(null, scoped_db, tableName, null);
+ } catch (Exception ex) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + ex.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + ex);
+ }
+ return rs;
+}
+
+ @Override protected String getCatalogName() {
+ return null;
+ }
+
+ @Override protected String getDatabaseName() {
+ return scoped_db;
Review Comment:
Thanks for explanation. This is a corner case and I agee with you that we can map that end database directly using a local connector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27597: Implement data connector for Hive to Hive federation over… [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #4720:
URL: https://github.com/apache/hive/pull/4720#discussion_r1348877014
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/AbstractJDBCConnectorProvider.java:
##########
@@ -234,9 +234,32 @@ protected boolean isClosed() {
return null;
}
- protected abstract ResultSet fetchTableMetadata(String tableName) throws MetaException;
+ protected ResultSet fetchTableMetadata(String tableName) throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(getCatalogName(), getDatabaseName(), null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
Review Comment:
nit
``
LOG.warn("Could not retrieve table names from remote datasource, cause: {}", sqle.getMessage());
``
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/AbstractJDBCConnectorProvider.java:
##########
@@ -234,9 +234,32 @@ protected boolean isClosed() {
return null;
}
- protected abstract ResultSet fetchTableMetadata(String tableName) throws MetaException;
+ protected ResultSet fetchTableMetadata(String tableName) throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(getCatalogName(), getDatabaseName(), null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ throw new MetaException("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
+ }
+ return rs;
+ }
- protected abstract ResultSet fetchTableNames() throws MetaException;
+ /**
+ * Returns a list of all table names from the remote database.
+ * @return List A collection of all the table names, null if there are no tables.
+ * @throws MetaException To indicate any failures with executing this API
+ */
+ protected ResultSet fetchTableNames() throws MetaException {
+ ResultSet rs = null;
+ try {
+ rs = getConnection().getMetaData().getTables(getCatalogName(), getDatabaseName(), null, new String[] { "TABLE" });
+ } catch (SQLException sqle) {
+ LOG.warn("Could not retrieve table names from remote datasource, cause:" + sqle.getMessage());
Review Comment:
change to
```
LOG.warn("Could not retrieve table names from remote datasource, cause: {}", sqle.getMessage());
```
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/AbstractJDBCConnectorProvider.java:
##########
@@ -415,4 +438,7 @@ protected Properties getConnectionProperties() {
connectorPropMap.forEach((k, v) -> connectionProperties.put(k, v));
return connectionProperties;
}
-}
\ No newline at end of file
+
+ @Override
+ protected String getDatasourceType() { return type; };
Review Comment:
I think there is a semi-colon after } which isn't required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27597: Implement data connector for Hive to Hive federation over… [hive]
Posted by "nrg4878 (via GitHub)" <gi...@apache.org>.
nrg4878 commented on code in PR #4720:
URL: https://github.com/apache/hive/pull/4720#discussion_r1348237467
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/HiveJDBCConnectorProvider.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class HiveJDBCConnectorProvider extends AbstractJDBCConnectorProvider {
+ private static Logger LOG = LoggerFactory.getLogger(HiveJDBCConnectorProvider.class);
+
+ private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver".intern();
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27597: Implement data connector for Hive to Hive federation over… [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4720:
URL: https://github.com/apache/hive/pull/4720#issuecomment-1750436704
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4720)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4720&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4720&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4720&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=CODE_SMELL) [31 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4720&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4720&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4720&metric=duplicated_lines_density&view=list) No Duplication information
 The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org