You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/10/17 06:36:56 UTC
[GitHub] [incubator-seatunnel] 531651225 opened a new pull request, #3116: [Feature][Connector-V2] Kylin source connector
531651225 opened a new pull request, #3116:
URL: https://github.com/apache/incubator-seatunnel/pull/3116
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
## Purpose of this pull request
add Kylin source Connector
refer to
- hive support datatype
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types
- kylin jdbc
https://kylin.apache.org/cn/docs31/howto/howto_jdbc.html
### tips
Tip: Since need to configure the kylin cube model and build the data in the kylin UI to complete the data preparation, it seems that you cannot add e2e to the program
https://github.com/apache/incubator-seatunnel/issues/3018
<!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
## Check list
* [x] Code changed are covered with tests, or it does not need tests for reason:
* [x] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] 531651225 commented on pull request #3116: [Feature][Connector-V2] Kylin source connector
Posted by GitBox <gi...@apache.org>.
531651225 commented on PR #3116:
URL: https://github.com/apache/incubator-seatunnel/pull/3116#issuecomment-1289895796
> Please add e2e test
Since need to configure the kylin cube model and build the data in the kylin UI to complete the data preparation, it seems that you cannot add e2e to the program. so I added a tips to describe this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] 531651225 commented on pull request #3116: [Feature][Connector-V2] Kylin source connector
Posted by GitBox <gi...@apache.org>.
531651225 commented on PR #3116:
URL: https://github.com/apache/incubator-seatunnel/pull/3116#issuecomment-1281816836
> Please add e2e in https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e
Since need to configure the kylin cube model and build the data in the kylin UI to complete the data preparation, it seems that you cannot add e2e to the program. so I added a tips to describe this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on pull request #3116: [Feature][Connector-V2] Kylin source connector
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3116:
URL: https://github.com/apache/incubator-seatunnel/pull/3116#issuecomment-1472934359
This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on a diff in pull request #3116: [Feature][Connector-V2] Kylin source connector
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on code in PR #3116:
URL: https://github.com/apache/incubator-seatunnel/pull/3116#discussion_r1010138981
##########
docs/en/connector-v2/source/Jdbc.md:
##########
@@ -100,6 +100,7 @@ there are some reference value for params above.
| oracle | oracle.jdbc.OracleDriver | jdbc:oracle:thin:@localhost:1521/xepdb1 | https://mvnrepository.com/artifact/com.oracle.database.jdbc/ojdbc8 |
| gbase8a | com.gbase.jdbc.Driver | jdbc:gbase://e2e_gbase8aDb:5258/test | https://www.gbase8.cn/wp-content/uploads/2020/10/gbase-connector-java-8.3.81.53-build55.5.7-bin_min_mix.jar |
| starrocks | com.mysql.cj.jdbc.Driver | jdbc:mysql://localhost:3306/test | https://mvnrepository.com/artifact/mysql/mysql-connector-java |
+| kylin | org.apache.kylin.jdbc.Driver | jdbc:kylin://localhost:7070/kylin_project_name | https://mvnrepository.com/artifact/org.apache.kylin/kylin-jdbc|
Review Comment:
Please add `changed log` reference https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/connector-v2/source/Redis.md#next-version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on pull request #3116: [Feature][Connector-V2] Kylin source connector
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on PR #3116:
URL: https://github.com/apache/incubator-seatunnel/pull/3116#issuecomment-1286515835
Please fix the CI problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ic4y commented on pull request #3116: [Feature][Connector-V2] Kylin source connector
Posted by GitBox <gi...@apache.org>.
ic4y commented on PR #3116:
URL: https://github.com/apache/incubator-seatunnel/pull/3116#issuecomment-1289904165
> > Please add e2e test
> > Since need to configure the kylin cube model and build the data in the kylin UI to complete the data preparation, it seems that you cannot add e2e to the program. so I added a tips to describe this
Got it, if e2e testing is very difficult. You can comment below your local testing process in the form of pictures, etc. This will make it easier to troubleshoot later if there is a problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] 531651225 commented on a diff in pull request #3116: [Feature][Connector-V2] Kylin source connector
Posted by GitBox <gi...@apache.org>.
531651225 commented on code in PR #3116:
URL: https://github.com/apache/incubator-seatunnel/pull/3116#discussion_r997720125
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/kylin/KylinTypeMapper.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.kylin;
+
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.DecimalType;
+import org.apache.seatunnel.api.table.type.LocalTimeType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.JdbcDialectTypeMapper;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSetMetaData;
+import java.sql.SQLException;
+
+public class KylinTypeMapper implements JdbcDialectTypeMapper {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KylinTypeMapper.class);
+
+ // ============================data types=====================
+
+ private static final String KYLIN_UNKNOWN = "UNKNOWN";
+ private static final String KYLIN_BOOLEAN = "BOOLEAN";
+
+ // -------------------------number----------------------------
+ private static final String KYLIN_TINYINT = "TINYINT";
+ private static final String KYLIN_SMALLINT = "SMALLINT";
+ private static final String KYLIN_INT = "INT";
+ private static final String KYLIN_BIGINT = "BIGINT";
+ private static final String KYLIN_DECIMAL = "DECIMAL";
+ private static final String KYLIN_FLOAT = "FLOAT";
+ private static final String KYLIN_DOUBLE = "DOUBLE";
+
+ // -------------------------string----------------------------
+ private static final String KYLIN_CHAR = "CHAR";
+ private static final String KYLIN_VARCHAR = "VARCHAR";
+ private static final String KYLIN_STRING = "STRING";
+
+ // ------------------------------time-------------------------
+ private static final String KYLIN_DATE = "DATE";
+ private static final String KYLIN_TIME = "TIME";
+ private static final String KYLIN_TIMESTAMP = "TIMESTAMP";
+
+ @SuppressWarnings("checkstyle:MagicNumber")
+ @Override
+ public SeaTunnelDataType<?> mapping(ResultSetMetaData metadata, int colIndex) throws SQLException {
+ String kylinType = metadata.getColumnTypeName(colIndex).toUpperCase();
+ int precision = metadata.getPrecision(colIndex);
+ int scale = metadata.getScale(colIndex);
+ switch (kylinType) {
+ case KYLIN_BOOLEAN:
+ return BasicType.BOOLEAN_TYPE;
+ case KYLIN_TINYINT:
Review Comment:
> We have `short` type maybe you can use it to represent `tinyint`
thinks your good advice，i have fixed it in new commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on a diff in pull request #3116: [Feature][Connector-V2] Kylin source connector
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on code in PR #3116:
URL: https://github.com/apache/incubator-seatunnel/pull/3116#discussion_r997675426
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/kylin/KylinTypeMapper.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.kylin;
+
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.DecimalType;
+import org.apache.seatunnel.api.table.type.LocalTimeType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.JdbcDialectTypeMapper;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSetMetaData;
+import java.sql.SQLException;
+
+public class KylinTypeMapper implements JdbcDialectTypeMapper {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KylinTypeMapper.class);
+
+ // ============================data types=====================
+
+ private static final String KYLIN_UNKNOWN = "UNKNOWN";
+ private static final String KYLIN_BOOLEAN = "BOOLEAN";
+
+ // -------------------------number----------------------------
+ private static final String KYLIN_TINYINT = "TINYINT";
+ private static final String KYLIN_SMALLINT = "SMALLINT";
+ private static final String KYLIN_INT = "INT";
+ private static final String KYLIN_BIGINT = "BIGINT";
+ private static final String KYLIN_DECIMAL = "DECIMAL";
+ private static final String KYLIN_FLOAT = "FLOAT";
+ private static final String KYLIN_DOUBLE = "DOUBLE";
+
+ // -------------------------string----------------------------
+ private static final String KYLIN_CHAR = "CHAR";
+ private static final String KYLIN_VARCHAR = "VARCHAR";
+ private static final String KYLIN_STRING = "STRING";
+
+ // ------------------------------time-------------------------
+ private static final String KYLIN_DATE = "DATE";
+ private static final String KYLIN_TIME = "TIME";
+ private static final String KYLIN_TIMESTAMP = "TIMESTAMP";
+
+ @SuppressWarnings("checkstyle:MagicNumber")
+ @Override
+ public SeaTunnelDataType<?> mapping(ResultSetMetaData metadata, int colIndex) throws SQLException {
+ String kylinType = metadata.getColumnTypeName(colIndex).toUpperCase();
+ int precision = metadata.getPrecision(colIndex);
+ int scale = metadata.getScale(colIndex);
+ switch (kylinType) {
+ case KYLIN_BOOLEAN:
+ return BasicType.BOOLEAN_TYPE;
+ case KYLIN_TINYINT:
Review Comment:
We have `short` type maybe you can use it to represent `tinyint`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] 531651225 commented on pull request #3116: [Feature][Connector-V2] Kylin source connector
Posted by GitBox <gi...@apache.org>.
531651225 commented on PR #3116:
URL: https://github.com/apache/incubator-seatunnel/pull/3116#issuecomment-1287039351
> Please fix the CI problems.
thinks, i have fix it. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] closed pull request #3116: [Feature][Connector-V2] Kylin source connector
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3116: [Feature][Connector-V2] Kylin source connector
URL: https://github.com/apache/incubator-seatunnel/pull/3116
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on pull request #3116: [Feature][Connector-V2] Kylin source connector
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on PR #3116:
URL: https://github.com/apache/incubator-seatunnel/pull/3116#issuecomment-1281755098
Please add e2e in https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ic4y commented on pull request #3116: [Feature][Connector-V2] Kylin source connector
Posted by GitBox <gi...@apache.org>.
ic4y commented on PR #3116:
URL: https://github.com/apache/incubator-seatunnel/pull/3116#issuecomment-1289893913
Please add e2e test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on pull request #3116: [Feature][Connector-V2] Kylin source connector
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3116:
URL: https://github.com/apache/incubator-seatunnel/pull/3116#issuecomment-1483626782
This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org