You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/05/24 10:49:26 UTC
[GitHub] [incubator-inlong] baomingyu opened a new pull request, #4357: [INLONG-4356][Manager]Support create postgres resource
baomingyu opened a new pull request, #4357:
URL: https://github.com/apache/incubator-inlong/pull/4357
Support create postgres resource
Fixes #4356
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] baomingyu commented on a diff in pull request #4357: [INLONG-4356][Manager] Support create postgres resource
Posted by GitBox <gi...@apache.org>.
baomingyu commented on code in PR #4357:
URL: https://github.com/apache/incubator-inlong/pull/4357#discussion_r881112573
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/postgres/PostgresJdbcUtils.java:
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.service.resource.postgres;
+
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.inlong.manager.common.pojo.sink.postgres.PostgresColumnInfo;
+import org.apache.inlong.manager.common.pojo.sink.postgres.PostgresTableInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utils for Postgres JDBC.
+ */
+public class PostgresJdbcUtils {
+
+ private static final String POSTGRES_DRIVER_CLASS = "org.postgresql.Driver";
+ private static final String COLUMN_LABEL = "TABLE_NAME";
+ private static final String POSTGRES_JDBC_PREFIX = "jdbc:postgresql";
+
+ private static final Logger LOG = LoggerFactory.getLogger(PostgresJdbcUtils.class);
+
+ /**
+ * Get Postgres connection from Postgres url and user
+ */
+ public static Connection getConnection(String url, String user, String password) throws Exception {
+ if (StringUtils.isBlank(url) || !url.startsWith(POSTGRES_JDBC_PREFIX)) {
+ throw new Exception("Postgres server URL was invalid, it should start with jdbc:postgresql");
+ }
+ Connection conn;
+ try {
+ Class.forName(POSTGRES_DRIVER_CLASS);
+ conn = DriverManager.getConnection(url, user, password);
+ } catch (Exception e) {
+ LOG.error("get postgres connection error, please check postgres jdbc url, username " +
+ "or password", e);
+ throw new Exception("get postgres connection error, please check jdbc url, username or password. "
+ + "other error msg: " + e.getMessage());
+ }
+
+ if (conn == null) {
+ throw new Exception("get postgres connection failed, please contact administrator");
+ }
+
+ LOG.info("get postgres connection success, url={}", url);
+ return conn;
+ }
+
+ /**
+ * Execute One Postgres Sql command
+ */
+ public static boolean executeCheckSql(String sql, String url, String user, String password) throws Exception {
+ try (Connection conn = getConnection(url, user, password)) {
+ Statement stmt = conn.createStatement();
+ LOG.info("execute sql [{}] success for url: {}", sql, url);
+ return stmt.execute(sql);
+ }
+ }
+
+ /**
+ * Execute One Postgres Sql command
+ */
+ public static ResultSet executeSql(String sql, String url, String user, String password) throws Exception {
Review Comment:
Please look at the implementation logic, which requires different return value types and different execution methods
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] gong commented on a diff in pull request #4357: [INLONG-4356][Manager] Support create postgres resource
Posted by GitBox <gi...@apache.org>.
gong commented on code in PR #4357:
URL: https://github.com/apache/incubator-inlong/pull/4357#discussion_r881639102
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/postgres/PostgresJdbcUtils.java:
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.service.resource.postgres;
+
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.inlong.manager.common.pojo.sink.postgres.PostgresColumnInfo;
+import org.apache.inlong.manager.common.pojo.sink.postgres.PostgresTableInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utils for Postgres JDBC.
+ */
+public class PostgresJdbcUtils {
+
+ private static final String POSTGRES_DRIVER_CLASS = "org.postgresql.Driver";
+ private static final String COLUMN_LABEL = "TABLE_NAME";
+ private static final String POSTGRES_JDBC_PREFIX = "jdbc:postgresql";
+
+ private static final Logger LOG = LoggerFactory.getLogger(PostgresJdbcUtils.class);
+
+ /**
+ * Get Postgres connection from Postgres url and user
+ */
+ public static Connection getConnection(String url, String user, String password)
+ throws Exception {
+ if (StringUtils.isBlank(url) || !url.startsWith(POSTGRES_JDBC_PREFIX)) {
+ throw new Exception(
+ "Postgres server URL was invalid, it should start with jdbc:postgresql");
+ }
+ Connection conn;
+ try {
+ Class.forName(POSTGRES_DRIVER_CLASS);
+ conn = DriverManager.getConnection(url, user, password);
+ } catch (Exception e) {
+ LOG.error("get postgres connection error, please check postgres jdbc url, username "
+ + "or password", e);
+ throw new Exception(
+ "get postgres connection error, please check jdbc url, username or password. "
Review Comment:
Content is same. May be declare string variable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] healchow merged pull request #4357: [INLONG-4356][Manager] Support create postgres resource
Posted by GitBox <gi...@apache.org>.
healchow merged PR #4357:
URL: https://github.com/apache/incubator-inlong/pull/4357
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] healchow commented on a diff in pull request #4357: [INLONG-4356][Manager] Support create postgres resource
Posted by GitBox <gi...@apache.org>.
healchow commented on code in PR #4357:
URL: https://github.com/apache/incubator-inlong/pull/4357#discussion_r880443423
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/postgres/PostgresJdbcUtils.java:
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.service.resource.postgres;
+
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.inlong.manager.common.pojo.sink.postgres.PostgresColumnInfo;
+import org.apache.inlong.manager.common.pojo.sink.postgres.PostgresTableInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utils for Postgres JDBC.
+ */
+public class PostgresJdbcUtils {
+
+ private static final String POSTGRES_DRIVER_CLASS = "org.postgresql.Driver";
+ private static final String COLUMN_LABEL = "TABLE_NAME";
+ private static final String POSTGRES_JDBC_PREFIX = "jdbc:postgresql";
+
+ private static final Logger LOG = LoggerFactory.getLogger(PostgresJdbcUtils.class);
+
+ /**
+ * Get Postgres connection from Postgres url and user
+ */
+ public static Connection getConnection(String url, String user, String password) throws Exception {
+ if (StringUtils.isBlank(url) || !url.startsWith(POSTGRES_JDBC_PREFIX)) {
+ throw new Exception("Postgres server URL was invalid, it should start with jdbc:postgresql");
+ }
+ Connection conn;
+ try {
+ Class.forName(POSTGRES_DRIVER_CLASS);
+ conn = DriverManager.getConnection(url, user, password);
+ } catch (Exception e) {
+ LOG.error("get postgres connection error, please check postgres jdbc url, username " +
+ "or password", e);
+ throw new Exception("get postgres connection error, please check jdbc url, username or password. "
+ + "other error msg: " + e.getMessage());
+ }
+
+ if (conn == null) {
+ throw new Exception("get postgres connection failed, please contact administrator");
+ }
+
+ LOG.info("get postgres connection success, url={}", url);
+ return conn;
+ }
+
+ /**
+ * Execute One Postgres Sql command
+ */
+ public static boolean executeCheckSql(String sql, String url, String user, String password) throws Exception {
+ try (Connection conn = getConnection(url, user, password)) {
+ Statement stmt = conn.createStatement();
+ LOG.info("execute sql [{}] success for url: {}", sql, url);
+ return stmt.execute(sql);
+ }
+ }
+
+ /**
+ * Execute One Postgres Sql command
+ */
+ public static ResultSet executeSql(String sql, String url, String user, String password) throws Exception {
Review Comment:
Why not reuse the `executeSql` and replace the `executeCheckSql`?
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/postgres/PostgresJdbcUtils.java:
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.service.resource.postgres;
+
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.inlong.manager.common.pojo.sink.postgres.PostgresColumnInfo;
+import org.apache.inlong.manager.common.pojo.sink.postgres.PostgresTableInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utils for Postgres JDBC.
+ */
+public class PostgresJdbcUtils {
+
+ private static final String POSTGRES_DRIVER_CLASS = "org.postgresql.Driver";
+ private static final String COLUMN_LABEL = "TABLE_NAME";
+ private static final String POSTGRES_JDBC_PREFIX = "jdbc:postgresql";
+
+ private static final Logger LOG = LoggerFactory.getLogger(PostgresJdbcUtils.class);
+
+ /**
+ * Get Postgres connection from Postgres url and user
+ */
+ public static Connection getConnection(String url, String user, String password) throws Exception {
+ if (StringUtils.isBlank(url) || !url.startsWith(POSTGRES_JDBC_PREFIX)) {
+ throw new Exception("Postgres server URL was invalid, it should start with jdbc:postgresql");
+ }
+ Connection conn;
+ try {
+ Class.forName(POSTGRES_DRIVER_CLASS);
+ conn = DriverManager.getConnection(url, user, password);
+ } catch (Exception e) {
+ LOG.error("get postgres connection error, please check postgres jdbc url, username " +
+ "or password", e);
+ throw new Exception("get postgres connection error, please check jdbc url, username or password. "
+ + "other error msg: " + e.getMessage());
+ }
+
+ if (conn == null) {
+ throw new Exception("get postgres connection failed, please contact administrator");
+ }
+
+ LOG.info("get postgres connection success, url={}", url);
+ return conn;
+ }
+
+ /**
+ * Execute One Postgres Sql command
+ */
+ public static boolean executeCheckSql(String sql, String url, String user, String password) throws Exception {
+ try (Connection conn = getConnection(url, user, password)) {
+ Statement stmt = conn.createStatement();
+ LOG.info("execute sql [{}] success for url: {}", sql, url);
+ return stmt.execute(sql);
+ }
+ }
+
+ /**
+ * Execute One Postgres Sql command
+ */
+ public static ResultSet executeSql(String sql, String url, String user, String password) throws Exception {
+ try (Connection conn = getConnection(url, user, password)) {
+ Statement stmt = conn.createStatement();
+ LOG.info("execute sql [{}] success for url: {}", sql, url);
+ return stmt.executeQuery(sql);
+ }
+ }
+
+ /**
+ * Execute Batch Postgres Sql commands
+ */
+ public static void executeSqlBatch(List<String> sql, String url, String user, String password) throws Exception {
+ try (Connection conn = getConnection(url, user, password)) {
+ Statement stmt = conn.createStatement();
+ for (String entry : sql) {
+ stmt.execute(entry);
+ }
+ LOG.info("execute sql [{}] success for url: {}", sql, url);
+ }
+ }
+
+ /**
+ * Create Postgres database
+ */
+ public static void createDb(String url, String user, String password, String dbName) throws Exception {
+ String checkDbSql = PostgresSqlBuilder.getCheckDatabase(dbName);
+ ResultSet resultSet = executeSql(checkDbSql, url, user, password);
+ if (resultSet != null) {
+ resultSet.next();
+ if (resultSet.getInt("count") == 0) {
+ String createDbSql = PostgresSqlBuilder.buildCreateDbSql(dbName);
+ executeCheckSql(createDbSql, url, user, password);
+ }
+ }
+ }
+
+ /**
+ * Create Postgres table
+ */
+ public static void createTable(String url, String user, String password,
+ PostgresTableInfo tableInfo) throws Exception {
+ String createTableSql = PostgresSqlBuilder.buildCreateTableSql(tableInfo);
+ PostgresJdbcUtils.executeCheckSql(createTableSql, url, user, password);
+ }
+
+ /**
+ * Get Postgres tables from the Postgres metadata
+ */
+ public static boolean checkTablesExist(String url, String user, String password,
+ String dbname,
+ String tableName
+ ) throws Exception {
+ ResultSet rs = null;
+ boolean result = false;
+ try (Connection conn = getConnection(url, user, password)){
+ DatabaseMetaData metaData = conn.getMetaData();
+ rs = metaData.getTables(conn.getCatalog(), "public", tableName, new String[]{
+ "TABLE"});
+ if (rs != null) {
+ rs.next();
+ result = (rs != null && rs.getRow() > 0 && tableName.equals(rs.getString(COLUMN_LABEL)));
+ LOG.info("dbname is {}/{}/{}", dbname, tableName, result);
Review Comment:
What does the `dbname is xxx` means?
Suggest change to `LOG.info("db {} table {} exist? {}", dbname, tableName, result);`.
##########
inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/pojo/sink/postgres/PostgresSinkDTO.java:
##########
@@ -28,6 +29,9 @@
import lombok.NoArgsConstructor;
import org.apache.inlong.manager.common.enums.ErrorCodeEnum;
import org.apache.inlong.manager.common.exceptions.BusinessException;
+import org.apache.inlong.manager.common.pojo.sink.ck.ClickHouseColumnInfo;
Review Comment:
Why add those unused imports here?
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/postgres/PostgresJdbcUtils.java:
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.service.resource.postgres;
+
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.inlong.manager.common.pojo.sink.postgres.PostgresColumnInfo;
+import org.apache.inlong.manager.common.pojo.sink.postgres.PostgresTableInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utils for Postgres JDBC.
+ */
+public class PostgresJdbcUtils {
+
+ private static final String POSTGRES_DRIVER_CLASS = "org.postgresql.Driver";
+ private static final String COLUMN_LABEL = "TABLE_NAME";
+ private static final String POSTGRES_JDBC_PREFIX = "jdbc:postgresql";
+
+ private static final Logger LOG = LoggerFactory.getLogger(PostgresJdbcUtils.class);
+
+ /**
+ * Get Postgres connection from Postgres url and user
+ */
+ public static Connection getConnection(String url, String user, String password) throws Exception {
+ if (StringUtils.isBlank(url) || !url.startsWith(POSTGRES_JDBC_PREFIX)) {
+ throw new Exception("Postgres server URL was invalid, it should start with jdbc:postgresql");
+ }
+ Connection conn;
+ try {
+ Class.forName(POSTGRES_DRIVER_CLASS);
+ conn = DriverManager.getConnection(url, user, password);
+ } catch (Exception e) {
+ LOG.error("get postgres connection error, please check postgres jdbc url, username " +
+ "or password", e);
+ throw new Exception("get postgres connection error, please check jdbc url, username or password. "
+ + "other error msg: " + e.getMessage());
+ }
+
+ if (conn == null) {
+ throw new Exception("get postgres connection failed, please contact administrator");
+ }
+
+ LOG.info("get postgres connection success, url={}", url);
+ return conn;
+ }
+
+ /**
+ * Execute One Postgres Sql command
+ */
+ public static boolean executeCheckSql(String sql, String url, String user, String password) throws Exception {
+ try (Connection conn = getConnection(url, user, password)) {
+ Statement stmt = conn.createStatement();
+ LOG.info("execute sql [{}] success for url: {}", sql, url);
+ return stmt.execute(sql);
+ }
+ }
+
+ /**
+ * Execute One Postgres Sql command
+ */
+ public static ResultSet executeSql(String sql, String url, String user, String password) throws Exception {
+ try (Connection conn = getConnection(url, user, password)) {
+ Statement stmt = conn.createStatement();
+ LOG.info("execute sql [{}] success for url: {}", sql, url);
+ return stmt.executeQuery(sql);
+ }
+ }
+
+ /**
+ * Execute Batch Postgres Sql commands
+ */
+ public static void executeSqlBatch(List<String> sql, String url, String user, String password) throws Exception {
+ try (Connection conn = getConnection(url, user, password)) {
+ Statement stmt = conn.createStatement();
+ for (String entry : sql) {
+ stmt.execute(entry);
+ }
+ LOG.info("execute sql [{}] success for url: {}", sql, url);
+ }
+ }
+
+ /**
+ * Create Postgres database
+ */
+ public static void createDb(String url, String user, String password, String dbName) throws Exception {
+ String checkDbSql = PostgresSqlBuilder.getCheckDatabase(dbName);
+ ResultSet resultSet = executeSql(checkDbSql, url, user, password);
+ if (resultSet != null) {
+ resultSet.next();
+ if (resultSet.getInt("count") == 0) {
+ String createDbSql = PostgresSqlBuilder.buildCreateDbSql(dbName);
+ executeCheckSql(createDbSql, url, user, password);
+ }
+ }
+ }
+
+ /**
+ * Create Postgres table
+ */
+ public static void createTable(String url, String user, String password,
+ PostgresTableInfo tableInfo) throws Exception {
+ String createTableSql = PostgresSqlBuilder.buildCreateTableSql(tableInfo);
+ PostgresJdbcUtils.executeCheckSql(createTableSql, url, user, password);
+ }
+
+ /**
+ * Get Postgres tables from the Postgres metadata
+ */
+ public static boolean checkTablesExist(String url, String user, String password,
+ String dbname,
+ String tableName
Review Comment:
Those lines can merge into one line, which may improve the readability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] healchow commented on a diff in pull request #4357: [INLONG-4356][Manager] Support create postgres resource
Posted by GitBox <gi...@apache.org>.
healchow commented on code in PR #4357:
URL: https://github.com/apache/incubator-inlong/pull/4357#discussion_r880468204
##########
inlong-manager/manager-service/src/test/java/org/apache/inlong/manager/service/core/sink/PostgresStreamSinkServiceTest.java:
##########
@@ -99,4 +109,53 @@ public void testGetAndUpdate() {
deletePostgresSink(postgresSinkId);
}
+// @Test
Review Comment:
Just comment out `@Test` and add the note: turned on when testing locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] gong commented on a diff in pull request #4357: [INLONG-4356][Manager] Support create postgres resource
Posted by GitBox <gi...@apache.org>.
gong commented on code in PR #4357:
URL: https://github.com/apache/incubator-inlong/pull/4357#discussion_r881639102
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/postgres/PostgresJdbcUtils.java:
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.service.resource.postgres;
+
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.inlong.manager.common.pojo.sink.postgres.PostgresColumnInfo;
+import org.apache.inlong.manager.common.pojo.sink.postgres.PostgresTableInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utils for Postgres JDBC.
+ */
+public class PostgresJdbcUtils {
+
+ private static final String POSTGRES_DRIVER_CLASS = "org.postgresql.Driver";
+ private static final String COLUMN_LABEL = "TABLE_NAME";
+ private static final String POSTGRES_JDBC_PREFIX = "jdbc:postgresql";
+
+ private static final Logger LOG = LoggerFactory.getLogger(PostgresJdbcUtils.class);
+
+ /**
+ * Get Postgres connection from Postgres url and user
+ */
+ public static Connection getConnection(String url, String user, String password)
+ throws Exception {
+ if (StringUtils.isBlank(url) || !url.startsWith(POSTGRES_JDBC_PREFIX)) {
+ throw new Exception(
+ "Postgres server URL was invalid, it should start with jdbc:postgresql");
+ }
+ Connection conn;
+ try {
+ Class.forName(POSTGRES_DRIVER_CLASS);
+ conn = DriverManager.getConnection(url, user, password);
+ } catch (Exception e) {
+ LOG.error("get postgres connection error, please check postgres jdbc url, username "
+ + "or password", e);
+ throw new Exception(
+ "get postgres connection error, please check jdbc url, username or password. "
Review Comment:
Content is same. May be throw exception is enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] gong commented on a diff in pull request #4357: [INLONG-4356][Manager] Support create postgres resource
Posted by GitBox <gi...@apache.org>.
gong commented on code in PR #4357:
URL: https://github.com/apache/incubator-inlong/pull/4357#discussion_r881684182
##########
inlong-manager/manager-service/src/test/java/org/apache/inlong/manager/service/core/sink/PostgresStreamSinkServiceTest.java:
##########
@@ -99,4 +107,53 @@ public void testGetAndUpdate() {
deletePostgresSink(postgresSinkId);
}
+ /**
+ * just using in local test
+ */
+ //@Test
+ public void testDbResource() {
Review Comment:
use @Ignore("just using in local test")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] gong commented on a diff in pull request #4357: [INLONG-4356][Manager] Support create postgres resource
Posted by GitBox <gi...@apache.org>.
gong commented on code in PR #4357:
URL: https://github.com/apache/incubator-inlong/pull/4357#discussion_r881662635
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/postgres/PostgresJdbcUtils.java:
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.service.resource.postgres;
+
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.inlong.manager.common.pojo.sink.postgres.PostgresColumnInfo;
+import org.apache.inlong.manager.common.pojo.sink.postgres.PostgresTableInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utils for Postgres JDBC.
+ */
+public class PostgresJdbcUtils {
+
+ private static final String POSTGRES_DRIVER_CLASS = "org.postgresql.Driver";
+ private static final String COLUMN_LABEL = "TABLE_NAME";
+ private static final String POSTGRES_JDBC_PREFIX = "jdbc:postgresql";
+
+ private static final Logger LOG = LoggerFactory.getLogger(PostgresJdbcUtils.class);
+
+ /**
+ * Get Postgres connection from Postgres url and user
+ */
+ public static Connection getConnection(String url, String user, String password)
+ throws Exception {
+ if (StringUtils.isBlank(url) || !url.startsWith(POSTGRES_JDBC_PREFIX)) {
+ throw new Exception(
+ "Postgres server URL was invalid, it should start with jdbc:postgresql");
+ }
+ Connection conn;
+ try {
+ Class.forName(POSTGRES_DRIVER_CLASS);
+ conn = DriverManager.getConnection(url, user, password);
+ } catch (Exception e) {
+ LOG.error("get postgres connection error, please check postgres jdbc url, username "
+ + "or password", e);
+ throw new Exception(
+ "get postgres connection error, please check jdbc url, username or password. "
+ + "other error msg: " + e.getMessage());
+ }
+
+ if (conn == null) {
+ throw new Exception("get postgres connection failed, please contact administrator");
+ }
+
+ LOG.info("get postgres connection success, url={}", url);
+ return conn;
+ }
+
+ /**
+ * Execute One Postgres Sql command
+ */
+ public static boolean executeCheckSql(String sql, String url, String user, String password)
+ throws Exception {
+ try (Connection conn = getConnection(url, user, password)) {
+ Statement stmt = conn.createStatement();
+ LOG.info("execute sql [{}] success for url: {}", sql, url);
+ return stmt.execute(sql);
+ }
+ }
+
+ /**
+ * Execute One Postgres Sql command
+ */
+ public static ResultSet executeSql(String sql, String url, String user, String password)
+ throws Exception {
+ try (Connection conn = getConnection(url, user, password)) {
+ Statement stmt = conn.createStatement();
+ LOG.info("execute sql [{}] success for url: {}", sql, url);
+ return stmt.executeQuery(sql);
+ }
+ }
+
+ /**
+ * Execute Batch Postgres Sql commands
+ */
+ public static void executeSqlBatch(List<String> sql, String url, String user, String password)
+ throws Exception {
+ try (Connection conn = getConnection(url, user, password)) {
+ Statement stmt = conn.createStatement();
+ for (String entry : sql) {
+ stmt.execute(entry);
+ }
+ LOG.info("execute sql [{}] success for url: {}", sql, url);
+ }
+ }
+
+ /**
+ * Create Postgres database
+ */
+ public static void createDb(String url, String user, String password, String dbName)
+ throws Exception {
+ String checkDbSql = PostgresSqlBuilder.getCheckDatabase(dbName);
+ ResultSet resultSet = executeSql(checkDbSql, url, user, password);
+ if (resultSet != null) {
+ resultSet.next();
+ if (resultSet.getInt("count") == 0) {
+ String createDbSql = PostgresSqlBuilder.buildCreateDbSql(dbName);
+ executeCheckSql(createDbSql, url, user, password);
Review Comment:
function name have ambiguity in here to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] gong commented on a diff in pull request #4357: [INLONG-4356][Manager] Support create postgres resource
Posted by GitBox <gi...@apache.org>.
gong commented on code in PR #4357:
URL: https://github.com/apache/incubator-inlong/pull/4357#discussion_r881647159
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/postgres/PostgresJdbcUtils.java:
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.service.resource.postgres;
+
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.inlong.manager.common.pojo.sink.postgres.PostgresColumnInfo;
+import org.apache.inlong.manager.common.pojo.sink.postgres.PostgresTableInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utils for Postgres JDBC.
+ */
+public class PostgresJdbcUtils {
+
+ private static final String POSTGRES_DRIVER_CLASS = "org.postgresql.Driver";
+ private static final String COLUMN_LABEL = "TABLE_NAME";
+ private static final String POSTGRES_JDBC_PREFIX = "jdbc:postgresql";
+
+ private static final Logger LOG = LoggerFactory.getLogger(PostgresJdbcUtils.class);
+
+ /**
+ * Get Postgres connection from Postgres url and user
+ */
+ public static Connection getConnection(String url, String user, String password)
+ throws Exception {
+ if (StringUtils.isBlank(url) || !url.startsWith(POSTGRES_JDBC_PREFIX)) {
+ throw new Exception(
+ "Postgres server URL was invalid, it should start with jdbc:postgresql");
+ }
+ Connection conn;
+ try {
+ Class.forName(POSTGRES_DRIVER_CLASS);
+ conn = DriverManager.getConnection(url, user, password);
+ } catch (Exception e) {
+ LOG.error("get postgres connection error, please check postgres jdbc url, username "
+ + "or password", e);
+ throw new Exception(
+ "get postgres connection error, please check jdbc url, username or password. "
+ + "other error msg: " + e.getMessage());
+ }
+
+ if (conn == null) {
+ throw new Exception("get postgres connection failed, please contact administrator");
+ }
+
+ LOG.info("get postgres connection success, url={}", url);
+ return conn;
+ }
+
+ /**
+ * Execute One Postgres Sql command
+ */
Review Comment:
I suggest describe should be different with `executeSql`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org