You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "changhuyan (via GitHub)" <gi...@apache.org> on 2023/04/20 02:43:21 UTC
[GitHub] [incubator-seatunnel] changhuyan opened a new pull request, #4626: [Feature][Jdbc-Connector] Add OceanBase Connector
changhuyan opened a new pull request, #4626:
URL: https://github.com/apache/incubator-seatunnel/pull/4626
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
## Purpose of this pull request
Add OceanBase Connector
## Check list
* [ ] Code changed are covered with tests, or it does not need tests for reason:
* [ ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
* [ ] If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to [connector-v2]
Examples are as follows：
env {
# You can set engine configuration here
job.name = "SeaTunnel"
spark.executor.instances = 1
spark.executor.cores = 1
spark.executor.memory = "1g"
spark.master = local
}
source {
jdbc{
# This is a example source plugin **only for test and demonstrate the feature source plugin**
url = "jdbc:oceanbase://xxxxx:2881/test"
driver = "com.alipay.oceanbase.jdbc.Driver"
user = "xxx"
password = "xxx"
query = "select id,name from test"
driver_type = "mysql"
}
}
transform {
}
sink {
jdbc {
url = "jdbc:oceanbase://xxx:2881/test"
driver = "com.alipay.oceanbase.jdbc.Driver"
user = "root"
password = "xxxxx"
driver_type = "oracle"
query = "insert into test3(id,name) values(?,?)"
}
}
3. Update [plugin-mapping.properties](https://github.com/apache/incubator-seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
4. Update the pom file of [seatunnel-dist](https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-dist/pom.xml)
* [ ] Update the [`release-note`](https://github.com/apache/incubator-seatunnel/blob/dev/release-note.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] whhe commented on a diff in pull request #4626: [Feature][Jdbc-Connector] Add OceanBase Connector
Posted by "whhe (via GitHub)" <gi...@apache.org>.
whhe commented on code in PR #4626:
URL: https://github.com/apache/seatunnel/pull/4626#discussion_r1243608747
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/oceanbase/OceanBaseMySqlDialectFactory.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.oceanbase;
+
+import org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.JdbcDialect;
+import org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.JdbcDialectFactory;
+import org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.mysql.MysqlDialect;
+
+import com.google.auto.service.AutoService;
+
+import java.util.Optional;
+
+@AutoService(JdbcDialectFactory.class)
+public class OceanBaseMySqlDialectFactory implements JdbcDialectFactory {
+ @Override
+ public boolean acceptsURL(String url, Optional<String> driverTye) {
+ return url.startsWith("jdbc:oceanbase:")
+ && driverTye.isPresent()
+ && driverTye.get().equalsIgnoreCase("mysql");
+ }
+
+ @Override
+ public JdbcDialect create() {
+ return new MysqlDialect();
Review Comment:
We can merge the two dialect classes of OceanBase and return the JdbcDialect by the `driverType` param here.
```java
public JdbcDialect create(@Nonnull String driverType) {
if ("mysql".equalsIgnoreCase(driverType)) {
return new MysqlDialect();
}
return new OracleDialect();
}
```
##########
seatunnel-dist/pom.xml:
##########
@@ -488,6 +489,12 @@
</dependency>
<!-- jdbc driver -->
+ <dependency>
+ <groupId>com.alipay.oceanbase</groupId>
+ <artifactId>oceanbase-client</artifactId>
+ <version>${oceanbase.version}</version>
+ <scope>provided</scope>
+ </dependency>
Review Comment:
Should not use the previous internal version, just keep it same with `seatunnel-connectors-v2/connector-jdbc/pom.xml`
##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-2/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcOceanbaseIT.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.jdbc;
+
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+
+import org.apache.commons.lang3.tuple.Pair;
+
+import org.junit.jupiter.api.Disabled;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.utility.DockerImageName;
+import org.testcontainers.utility.DockerLoggerFactory;
+
+import com.google.common.collect.Lists;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+@Disabled("Disabled because it needs user's personal oceanbase account to run this test!")
+public class JdbcOceanbaseIT extends AbstractJdbcIT {
+ private static final String OCEANBASE_IMAGE = "shihd/oceanbase:1.0";
Review Comment:
You can use the docker image `oceanbase/oceanbase-ce:4.0.0.0` with user `root` and empty password.
##########
seatunnel-connectors-v2/connector-jdbc/pom.xml:
##########
@@ -129,6 +130,12 @@
<version>${vertica.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>com.oceanbase</groupId>
+ <artifactId>oceanbase-client</artifactId>
+ <version>${oceanbase.version}</version>
+ <scope>provided</scope>
+ </dependency>
Review Comment:
It should not be imported directly due to the lgpl license https://github.com/oceanbase/obconnector-j
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/JdbcDialectFactory.java:
##########
@@ -33,7 +35,7 @@ public interface JdbcDialectFactory {
* @return <code>true</code> if this dialect understands the given URL; <code>false</code>
* otherwise.
*/
- boolean acceptsURL(String url);
Review Comment:
I think the `driverType` param shoud be passed in the create method below, and we can use a new default method to keep the other dialects unchanged.
```java
default JdbcDialect create(String driverType) {
return create();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on a diff in pull request #4626: [Feature][Jdbc-Connector] Add OceanBase Connector
Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 commented on code in PR #4626:
URL: https://github.com/apache/incubator-seatunnel/pull/4626#discussion_r1172027399
##########
seatunnel-examples/seatunnel-engine-examples/src/main/java/org/apache/seatunnel/example/engine/SeaTunnelEngineExample.java:
##########
@@ -31,7 +31,7 @@ public class SeaTunnelEngineExample {
public static void main(String[] args)
throws FileNotFoundException, URISyntaxException, CommandException {
- String configurePath = args.length > 0 ? args[0] : "/examples/fake_to_console.conf";
+ String configurePath = args.length > 0 ? args[0] : "/examples/oceabase.conf";
Review Comment:
revert
##########
seatunnel-examples/seatunnel-spark-connector-v2-example/src/main/java/org/apache/seatunnel/example/spark/v2/SeaTunnelApiExample.java:
##########
@@ -26,7 +26,7 @@ public class SeaTunnelApiExample {
public static void main(String[] args)
throws FileNotFoundException, URISyntaxException, CommandException {
- String configurePath = args.length > 0 ? args[0] : "/examples/spark.batch.conf";
+ String configurePath = args.length > 0 ? args[0] : "/examples/oceabase.conf";
Review Comment:
revert
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] EricJoy2048 closed pull request #4626: [Feature][Jdbc-Connector] Add OceanBase Connector
Posted by "EricJoy2048 (via GitHub)" <gi...@apache.org>.
EricJoy2048 closed pull request #4626: [Feature][Jdbc-Connector] Add OceanBase Connector
URL: https://github.com/apache/seatunnel/pull/4626
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on a diff in pull request #4626: [Feature][Jdbc-Connector] Add OceanBase Connector
Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 commented on code in PR #4626:
URL: https://github.com/apache/incubator-seatunnel/pull/4626#discussion_r1178556896
##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-2/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcOceanbaseIT.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.jdbc;
+
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+
+import org.apache.commons.lang3.tuple.Pair;
+
+import org.junit.jupiter.api.Disabled;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.utility.DockerImageName;
+import org.testcontainers.utility.DockerLoggerFactory;
+
+import com.google.common.collect.Lists;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+@Disabled("Disabled because it needs user's personal oceanbase account to run this test!")
Review Comment:
Use docker deploy ob server
https://en.oceanbase.com/docs/community-observer-en-10000000000829647#Use%20Docker%20to%20deploy%20OceanBase%20Database
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] changhuyan commented on a diff in pull request #4626: [Feature][Jdbc-Connector] Add OceanBase Connector
Posted by "changhuyan (via GitHub)" <gi...@apache.org>.
changhuyan commented on code in PR #4626:
URL: https://github.com/apache/incubator-seatunnel/pull/4626#discussion_r1173545843
##########
seatunnel-examples/seatunnel-spark-connector-v2-example/src/main/java/org/apache/seatunnel/example/spark/v2/SeaTunnelApiExample.java:
##########
@@ -26,7 +26,7 @@ public class SeaTunnelApiExample {
public static void main(String[] args)
throws FileNotFoundException, URISyntaxException, CommandException {
- String configurePath = args.length > 0 ? args[0] : "/examples/spark.batch.conf";
+ String configurePath = args.length > 0 ? args[0] : "/examples/fake_to_console.conf";
Review Comment:
ok
##########
seatunnel-connectors-v2/connector-jdbc/pom.xml:
##########
@@ -144,7 +151,10 @@
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
-
+ <dependency>
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on pull request #4626: [Feature][Jdbc-Connector] Add OceanBase Connector
Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 commented on PR #4626:
URL: https://github.com/apache/incubator-seatunnel/pull/4626#issuecomment-1515673067
please check ci error
https://github.com/apache/incubator-seatunnel/actions/runs/4749964848/jobs/8438153776?pr=4626
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on a diff in pull request #4626: [Feature][Jdbc-Connector] Add OceanBase Connector
Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 commented on code in PR #4626:
URL: https://github.com/apache/incubator-seatunnel/pull/4626#discussion_r1173374939
##########
seatunnel-examples/seatunnel-spark-connector-v2-example/src/main/java/org/apache/seatunnel/example/spark/v2/SeaTunnelApiExample.java:
##########
@@ -26,7 +26,7 @@ public class SeaTunnelApiExample {
public static void main(String[] args)
throws FileNotFoundException, URISyntaxException, CommandException {
- String configurePath = args.length > 0 ? args[0] : "/examples/spark.batch.conf";
+ String configurePath = args.length > 0 ? args[0] : "/examples/fake_to_console.conf";
Review Comment:
revert
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/oceanbase/OceanBaseMySqlDialectFactory.java:
##########
@@ -0,0 +1,22 @@
+package org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.oceanbase;
+
+import java.util.Optional;
+import com.google.auto.service.AutoService;
+import org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.JdbcDialect;
+import org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.JdbcDialectFactory;
+import org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.mysql.MysqlDialect;
+
+@AutoService(JdbcDialectFactory.class)
+public class OceanBaseMySqlDialectFactory implements JdbcDialectFactory {
+ @Override
+ public boolean acceptsURL(String url, Optional<String> driverTye) {
+ return url.startsWith("jdbc:oceanbase:")
+ && driverTye.isPresent()
+ && driverTye.get().equalsIgnoreCase("mysql");
Review Comment:
add example into docs
##########
seatunnel-connectors-v2/connector-jdbc/pom.xml:
##########
@@ -144,7 +151,10 @@
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
-
+ <dependency>
Review Comment:
add this drive into：
https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-dist/pom.xml#L558
https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-dist/src/main/assembly/assembly-bin-ci.xml#L195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on pull request #4626: [Feature][Jdbc-Connector] Add OceanBase Connector
Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 commented on PR #4626:
URL: https://github.com/apache/incubator-seatunnel/pull/4626#issuecomment-1518669802
1. check ci
2. add e2e testcase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] changhuyan commented on pull request #4626: [Feature][Jdbc-Connector] Add OceanBase Connector
Posted by "changhuyan (via GitHub)" <gi...@apache.org>.
changhuyan commented on PR #4626:
URL: https://github.com/apache/seatunnel/pull/4626#issuecomment-1609374210
Yes, there's a better way. You can change it.
------------------ 原始邮件 ------------------
发件人: "apache/seatunnel" ***@***.***>;
发送时间: 2023年6月27日(星期二) 晚上8:04
***@***.***>;
***@***.******@***.***>;
主题: Re: [apache/seatunnel] [Feature][Jdbc-Connector] Add OceanBase Connector (PR #4626)
@whhe requested changes on this pull request.
In seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/JdbcDialectFactory.java:
> @@ -33,7 +35,7 @@ public interface JdbcDialectFactory { * @return <code>true</code> if this dialect understands the given URL; <code>false</code> * otherwise. */ - boolean acceptsURL(String url);
I think the driverType param shoud be passed in the create method below, and we can use a new default method to keep the other dialects unchanged.
default JdbcDialect create(String driverType) { return create(); }
In seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/oceanbase/OceanBaseMySqlDialectFactory.java:
> +import com.google.auto.service.AutoService; + +import java.util.Optional; + ***@***.***(JdbcDialectFactory.class) +public class OceanBaseMySqlDialectFactory implements JdbcDialectFactory { + @Override + public boolean acceptsURL(String url, Optional<String> driverTye) { + return url.startsWith("jdbc:oceanbase:") + && driverTye.isPresent() + && driverTye.get().equalsIgnoreCase("mysql"); + } + + @Override + public JdbcDialect create() { + return new MysqlDialect();
We can merge the two dialect classes of OceanBase and return the JdbcDialect by the driverType param here.
public JdbcDialect ***@***.*** String driverType) { if ("mysql".equalsIgnoreCase(driverType)) { return new MysqlDialect(); } return new OracleDialect(); }
In seatunnel-connectors-v2/connector-jdbc/pom.xml:
> @@ -129,6 +130,12 @@ <version>${vertica.version}</version> <scope>provided</scope> </dependency> + <dependency> + <groupId>com.oceanbase</groupId> + <artifactId>oceanbase-client</artifactId> + <version>${oceanbase.version}</version> + <scope>provided</scope> + </dependency>
It should not be imported directly due to the lgpl license https://github.com/oceanbase/obconnector-j
In seatunnel-dist/pom.xml:
> @@ -488,6 +489,12 @@ </dependency> <!-- jdbc driver --> + <dependency> + <groupId>com.alipay.oceanbase</groupId> + <artifactId>oceanbase-client</artifactId> + <version>${oceanbase.version}</version> + <scope>provided</scope> + </dependency>
Should not use the previous internal version, just keep it same with seatunnel-connectors-v2/connector-jdbc/pom.xml
In seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-2/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcOceanbaseIT.java:
> +import org.junit.jupiter.api.Disabled; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.output.Slf4jLogConsumer; +import org.testcontainers.utility.DockerImageName; +import org.testcontainers.utility.DockerLoggerFactory; + +import com.google.common.collect.Lists; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + ***@***.***("Disabled because it needs user's personal oceanbase account to run this test!") +public class JdbcOceanbaseIT extends AbstractJdbcIT { + private static final String OCEANBASE_IMAGE = "shihd/oceanbase:1.0";
You can use the docker image oceanbase/oceanbase-ce:4.0.0.0 with user root and empty password.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] whhe commented on pull request #4626: [Feature][Jdbc-Connector] Add OceanBase Connector
Posted by "whhe (via GitHub)" <gi...@apache.org>.
whhe commented on PR #4626:
URL: https://github.com/apache/seatunnel/pull/4626#issuecomment-1609396147
@changhuyan Thanks for your contributation! I left some comments, and you can touch me at any time if you have any ideas on them.
For the reference to `oceanbase-client`, I think it may be an appropriate method to guide users to download it manually through the document, but I'm not sure how this affects the integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] NickYoungPeng commented on pull request #4626: [Feature][Jdbc-Connector] Add OceanBase Connector
Posted by "NickYoungPeng (via GitHub)" <gi...@apache.org>.
NickYoungPeng commented on PR #4626:
URL: https://github.com/apache/seatunnel/pull/4626#issuecomment-1565926424
why not use mysql connector
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on a diff in pull request #4626: [Feature][Jdbc-Connector] Add OceanBase Connector
Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 commented on code in PR #4626:
URL: https://github.com/apache/incubator-seatunnel/pull/4626#discussion_r1175954125
##########
docs/en/connector-v2/sink/Jdbc.md:
##########
@@ -33,6 +33,7 @@ support `Xa transactions`. You can set `is_exactly_once=true` to enable it.
| user | String | No | - |
| password | String | No | - |
| query | String | No | - |
+| driver_type | String | No | - |
Review Comment:
add appendix & axample
https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/connector-v2/source/Jdbc.md#appendix
##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-2/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcOceanbaseIT.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.jdbc;
+
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+
+import org.apache.commons.lang3.tuple.Pair;
+
+import org.junit.jupiter.api.Disabled;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.utility.DockerImageName;
+import org.testcontainers.utility.DockerLoggerFactory;
+
+import com.google.common.collect.Lists;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+@Disabled("Disabled because it needs user's personal oceanbase account to run this test!")
+public class JdbcOceanbaseIT extends AbstractJdbcIT {
+ private static final String OCEANBASE_IMAGE = "shihd/oceanbase:1.0";
+ private static final String OCEANBASE_CONTAINER_HOST = "spark_e2e_oceanbase";
+ private static final String OCEANBASE_DATABASE = "testdb";
+
+ private static final String OCEANBASE_SOURCE = "people";
+ private static final String OCEANBASE_SINK = "test3";
+
+ private static final String OCEANBASE_USERNAME = "root";
+ private static final String OCEANBASE_PASSWORD = "abcd";
+ private static final int OCEANBASE_CONTAINER_PORT = 2881;
+ private static final String OCEANBASE_URL = "jdbc:oceanbase://" + HOST + ":%s/%s";
+
+ private static final String DRIVER_CLASS = "com.alipay.oceanbase.jdbc.Driver";
+
+ private static final List<String> CONFIG_FILE =
+ Lists.newArrayList("/jdbc_oceanbase_mysql_source_and_oceanbase_oracle_sink.conf");
+
+ private static final String CREATE_SQL =
+ "CREATE TABLE `people` (\n"
+ + " `age` varchar(255) DEFAULT NULL,\n"
+ + " `name` varchar(255) DEFAULT NULL\n"
Review Comment:
Test all datatypes
##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-2/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcOceanbaseIT.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.jdbc;
+
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+
+import org.apache.commons.lang3.tuple.Pair;
+
+import org.junit.jupiter.api.Disabled;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.utility.DockerImageName;
+import org.testcontainers.utility.DockerLoggerFactory;
+
+import com.google.common.collect.Lists;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+@Disabled("Disabled because it needs user's personal oceanbase account to run this test!")
+public class JdbcOceanbaseIT extends AbstractJdbcIT {
+ private static final String OCEANBASE_IMAGE = "shihd/oceanbase:1.0";
+ private static final String OCEANBASE_CONTAINER_HOST = "spark_e2e_oceanbase";
+ private static final String OCEANBASE_DATABASE = "testdb";
+
+ private static final String OCEANBASE_SOURCE = "people";
+ private static final String OCEANBASE_SINK = "test3";
+
+ private static final String OCEANBASE_USERNAME = "root";
+ private static final String OCEANBASE_PASSWORD = "abcd";
+ private static final int OCEANBASE_CONTAINER_PORT = 2881;
+ private static final String OCEANBASE_URL = "jdbc:oceanbase://" + HOST + ":%s/%s";
+
+ private static final String DRIVER_CLASS = "com.alipay.oceanbase.jdbc.Driver";
+
+ private static final List<String> CONFIG_FILE =
+ Lists.newArrayList("/jdbc_oceanbase_mysql_source_and_oceanbase_oracle_sink.conf");
+
+ private static final String CREATE_SQL =
+ "CREATE TABLE `people` (\n"
+ + " `age` varchar(255) DEFAULT NULL,\n"
+ + " `name` varchar(255) DEFAULT NULL\n"
+ + ") ENGINE=InnoDB DEFAULT CHARSET=utf8";
+
+ @Override
+ JdbcCase getJdbcCase() {
+ Map<String, String> containerEnv = new HashMap<>();
+ String jdbcUrl = String.format(OCEANBASE_URL, OCEANBASE_CONTAINER_PORT, OCEANBASE_DATABASE);
+ Pair<String[], List<SeaTunnelRow>> testDataSet = initTestData();
+ String[] fieldNames = testDataSet.getKey();
+
+ String insertSql = insertTable(OCEANBASE_DATABASE, OCEANBASE_SOURCE, fieldNames);
+
+ return JdbcCase.builder()
+ .dockerImage(OCEANBASE_IMAGE)
+ .networkAliases(OCEANBASE_CONTAINER_HOST)
+ .containerEnv(containerEnv)
+ .driverClass(DRIVER_CLASS)
+ .host(HOST)
+ .port(OCEANBASE_CONTAINER_PORT)
+ .localPort(OCEANBASE_CONTAINER_PORT)
+ .jdbcTemplate(OCEANBASE_URL)
+ .jdbcUrl(jdbcUrl)
+ .userName(OCEANBASE_USERNAME)
+ .password(OCEANBASE_PASSWORD)
+ .database(OCEANBASE_DATABASE)
+ .sourceTable(OCEANBASE_SOURCE)
+ .sinkTable(OCEANBASE_SINK)
+ .createSql(CREATE_SQL)
+ .configFile(CONFIG_FILE)
+ .insertSql(insertSql)
+ .testData(testDataSet)
+ .build();
+ }
+
+ @Override
+ void compareResult() {}
Review Comment:
implement this logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] changhuyan commented on pull request #4626: [Feature][Jdbc-Connector] Add OceanBase Connector
Posted by "changhuyan (via GitHub)" <gi...@apache.org>.
changhuyan commented on PR #4626:
URL: https://github.com/apache/seatunnel/pull/4626#issuecomment-1566377082
Because OceanBase databases are used in two ways, namely mysql and oracle.
------------------ 原始邮件 ------------------
发件人: "apache/seatunnel" ***@***.***>;
发送时间: 2023年5月28日(星期天) 下午2:20
***@***.***>;
***@***.******@***.***>;
主题: Re: [apache/seatunnel] [Feature][Jdbc-Connector] Add OceanBase Connector (PR #4626)
why not use mysql connector
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on pull request #4626: [Feature][Jdbc-Connector] Add OceanBase Connector
Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 commented on PR #4626:
URL: https://github.com/apache/incubator-seatunnel/pull/4626#issuecomment-1517351575
please check ci error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] whhe commented on pull request #4626: [Feature][Jdbc-Connector] Add OceanBase Connector
Posted by "whhe (via GitHub)" <gi...@apache.org>.
whhe commented on PR #4626:
URL: https://github.com/apache/seatunnel/pull/4626#issuecomment-1610610259
> Yes, there's a better way. You can change it.
> […](#)
> ------------------ 原始邮件 ------------------ 发件人: "apache/seatunnel" ***@***.***>; 发送时间: 2023年6月27日(星期二) 晚上8:04 ***@***.***>; ***@***.******@***.***>; 主题: Re: [apache/seatunnel] [Feature][Jdbc-Connector] Add OceanBase Connector (PR #4626) @whhe requested changes on this pull request. In seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/JdbcDialectFactory.java: > @@ -33,7 +35,7 @@ public interface JdbcDialectFactory { * @return <code>true</code> if this dialect understands the given URL; <code>false</code> * otherwise. */ - boolean acceptsURL(String url); I think the driverType param shoud be passed in the create method below, and we can use a new default method to keep the other dialects unchanged. default JdbcDialect create(String driverType) { return create(); } In seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/
seatunnel/connectors/seatunnel/jdbc/internal/dialect/oceanbase/OceanBaseMySqlDialectFactory.java: > +import com.google.auto.service.AutoService; + +import java.util.Optional; + ***@***.***(JdbcDialectFactory.class) +public class OceanBaseMySqlDialectFactory implements JdbcDialectFactory { + @OverRide + public boolean acceptsURL(String url, Optional<String> driverTye) { + return url.startsWith("jdbc:oceanbase:") + && driverTye.isPresent() + && driverTye.get().equalsIgnoreCase("mysql"); + } + + @OverRide + public JdbcDialect create() { + return new MysqlDialect(); We can merge the two dialect classes of OceanBase and return the JdbcDialect by the driverType param here. public JdbcDialect ***@***.*** String driverType) { if ("mysql".equalsIgnoreCase(driverType)) { return new MysqlDialect(); } return new OracleDialect(); } In seatunnel-connectors-v2/connector-jdbc/pom.xml: > @@ -129,6 +130,12 @@ <version>${vertica.version}</version> <scope>provided</sco
pe> </dependency> + <dependency> + <groupId>com.oceanbase</groupId> + <artifactId>oceanbase-client</artifactId> + <version>${oceanbase.version}</version> + <scope>provided</scope> + </dependency> It should not be imported directly due to the lgpl license https://github.com/oceanbase/obconnector-j In seatunnel-dist/pom.xml: > @@ -488,6 +489,12 @@ </dependency> <!-- jdbc driver --> + <dependency> + <groupId>com.alipay.oceanbase</groupId> + <artifactId>oceanbase-client</artifactId> + <version>${oceanbase.version}</version> + <scope>provided</scope> + </dependency> Should not use the previous internal version, just keep it same with seatunnel-connectors-v2/connector-jdbc/pom.xml In seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-2/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcOceanbaseIT.java: > +import org.junit.jupiter.api.Disabled; +import org.testconta
iners.containers.GenericContainer; +import org.testcontainers.containers.output.Slf4jLogConsumer; +import org.testcontainers.utility.DockerImageName; +import org.testcontainers.utility.DockerLoggerFactory; + +import com.google.common.collect.Lists; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + ***@***.***("Disabled because it needs user's personal oceanbase account to run this test!") +public class JdbcOceanbaseIT extends AbstractJdbcIT { + private static final String OCEANBASE_IMAGE = "shihd/oceanbase:1.0"; You can use the docker image oceanbase/oceanbase-ce:4.0.0.0 with user root and empty password. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>
Do you have time to complete this work recently? If not, we can also find other developers in the community to continue development based on your work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org