You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "lvshaokang (via GitHub)" <gi...@apache.org> on 2023/03/08 08:22:55 UTC
[GitHub] [incubator-seatunnel] lvshaokang opened a new pull request, #4304: [Feature][Connector-V2][GoogleFirestore] Support GoogleFirestore Sink
lvshaokang opened a new pull request, #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
## Purpose of this pull request
<!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
Support google firestore sink.
## Check list
* [ ] Code changed are covered with tests, or it does not need tests for reason:
* [ ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
* [ ] If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/incubator-seatunnel/tree/dev/docs/en/connector-v2)
2. Update [plugin-mapping.properties](https://github.com/apache/incubator-seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
3. Update the pom file of [seatunnel-dist](https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-dist/pom.xml)
* [ ] Update the [`release-note`](https://github.com/apache/incubator-seatunnel/blob/dev/release-note.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lvshaokang commented on pull request #4304: [Feature][Connector][GoogleFirestore-Sink] Support GoogleFirestore Sink
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on PR #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304#issuecomment-1527013861
> @lvshaokang Suggest adding the [new version documentation](https://github.com/apache/incubator-seatunnel/issues/4544) for GoogleFirestore Sink in subsequent PR.Or create a related issue
@ic4y ok, i will supply new doc follow the rules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on pull request #4304: [Feature][Connector-V2][GoogleFirestore] Support GoogleFirestore Sink
Posted by "EricJoy2048 (via GitHub)" <gi...@apache.org>.
EricJoy2048 commented on PR #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304#issuecomment-1492860077
I have no problem, @liugddx @TaoZex @TyrantLucifer PTAL again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lvshaokang commented on a diff in pull request #4304: [Feature][Connector][GoogleFirestore-Sink] Support GoogleFirestore Sink
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on code in PR #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304#discussion_r1174446893
##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-google-firestore-e2e/src/test/java/org.apache.seatunnel.e2e.connector.google.firestore/GoogleFirestoreIT.java:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.e2e.connector.google.firestore;
+
+import org.apache.seatunnel.connectors.seatunnel.google.firestore.config.FirestoreConfig;
+import org.apache.seatunnel.e2e.common.TestResource;
+import org.apache.seatunnel.e2e.common.TestSuiteBase;
+import org.apache.seatunnel.e2e.common.container.TestContainer;
+import org.apache.seatunnel.e2e.common.util.ContainerUtil;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.TestTemplate;
+import org.testcontainers.containers.Container;
+
+import com.google.api.core.ApiFuture;
+import com.google.auth.oauth2.GoogleCredentials;
+import com.google.cloud.firestore.CollectionReference;
+import com.google.cloud.firestore.Firestore;
+import com.google.cloud.firestore.FirestoreOptions;
+import com.google.cloud.firestore.QueryDocumentSnapshot;
+import com.google.cloud.firestore.QuerySnapshot;
+import com.typesafe.config.Config;
+import com.typesafe.config.ConfigFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.util.Base64;
+import java.util.List;
+
+@Disabled("Disabled because it needs google firestore database to run this test")
+public class GoogleFirestoreIT extends TestSuiteBase implements TestResource {
+
+ private static final String FIRESTORE_CONF_FILE = "/firestore/fake_to_google_firestore.conf";
+
+ private String projectId;
+ private String collection;
+ private String credentials;
+ private Firestore db;
+ private CollectionReference collectionReference;
+
+ @BeforeAll
+ @Override
+ public void startUp() throws Exception {
+ initFirestoreConfig();
+ FirestoreOptions firestoreOptions =
+ FirestoreOptions.getDefaultInstance()
+ .toBuilder()
+ .setProjectId(projectId)
+ .setCredentials(
+ GoogleCredentials.fromStream(
+ new ByteArrayInputStream(
+ Base64.getDecoder().decode(credentials))))
+ .build();
+ this.db = firestoreOptions.getService();
+ this.collectionReference = db.collection(collection);
+ }
+
+ private void initFirestoreConfig() {
+ File file = ContainerUtil.getResourcesFile(FIRESTORE_CONF_FILE);
+ Config config = ConfigFactory.parseFile(file);
+ Config firestoreConfig = config.getConfig("sink").getConfig("GoogleFirestore");
+ this.projectId = firestoreConfig.getString(FirestoreConfig.PROJECT_ID.key());
+ this.collection = firestoreConfig.getString(FirestoreConfig.COLLECTION.key());
+ this.credentials = firestoreConfig.getString(FirestoreConfig.CREDENTIALS.key());
+ }
+
+ @AfterAll
+ @Override
+ public void tearDown() throws Exception {
+ if (db != null) {
+ db.close();
+ }
+ }
+
+ @TestTemplate
+ public void testGoogleFirestore(TestContainer container) throws Exception {
+ Container.ExecResult execResult = container.executeJob(FIRESTORE_CONF_FILE);
+ Assertions.assertEquals(0, execResult.getExitCode());
+
+ List<QueryDocumentSnapshot> documents = readSinkDataset();
+ Assertions.assertTrue(documents.size() > 1);
+ Assertions.assertEquals(15, documents.get(0).getData().size());
Review Comment:
@ic4y PTAL. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lvshaokang commented on a diff in pull request #4304: [Feature][Connector-V2][GoogleFirestore] Support GoogleFirestore Sink
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on code in PR #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304#discussion_r1136508747
##########
seatunnel-connectors-v2/connector-google-firestore/pom.xml:
##########
@@ -0,0 +1,90 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.seatunnel</groupId>
+ <artifactId>seatunnel-connectors-v2</artifactId>
+ <version>${revision}</version>
+ </parent>
+
+ <artifactId>connector-google-firestore</artifactId>
+ <name>SeaTunnel : Connectors V2 : Google Firestore</name>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.seatunnel</groupId>
+ <artifactId>connector-common</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>com.google.cloud</groupId>
+ <artifactId>google-cloud-firestore</artifactId>
+ <version>3.7.10</version>
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] liugddx commented on pull request #4304: [Feature][Connector-V2][GoogleFirestore] Support GoogleFirestore Sink
Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on PR #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304#issuecomment-1477368571
you also should update release-note https://github.com/apache/incubator-seatunnel/blob/dev/release-note.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on a diff in pull request #4304: [Feature][Connector-V2][GoogleFirestore] Support GoogleFirestore Sink
Posted by "EricJoy2048 (via GitHub)" <gi...@apache.org>.
EricJoy2048 commented on code in PR #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304#discussion_r1150025187
##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-google-firestore-e2e/src/test/resources/firestore/fake_to_google_firestore.conf:
##########
@@ -0,0 +1,66 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+######
+###### This config file is a demonstration of streaming processing in seatunnel config
+######
+
+env {
+ execution.parallelism = 1
+ job.mode = "BATCH"
+}
+
+source {
+ # This is a example source plugin **only for test and demonstrate the feature source plugin**
+ FakeSource {
+ schema = {
+ fields {
+ c_map = "map<string, string>"
+ c_array = "array<int>"
+ c_string = string
+ c_boolean = boolean
+ c_tinyint = tinyint
+ c_smallint = smallint
+ c_int = int
+ c_bigint = bigint
+ c_float = float
+ c_double = double
+ c_decimal = "decimal(30, 8)"
+ c_null = "null"
+ c_bytes = bytes
+ c_date = date
+ c_timestamp = timestamp
+ }
+ }
+ result_table_name = "fake"
+ }
+}
+
+transform {
Review Comment:
Remove this transform if it is unnecessary.
##########
seatunnel-connectors-v2/connector-google-firestore/src/main/java/org/apache/seatunnel/connectors/seatunnel/google/firestore/exception/FirestoreConnectorErrorCode.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.google.firestore.exception;
+
+import org.apache.seatunnel.common.exception.SeaTunnelErrorCode;
+
+public enum FirestoreConnectorErrorCode implements SeaTunnelErrorCode {
+ CLOSE_CLIENT_FAILED("FIRESTORE-01", "Close Firestore client failed");
Review Comment:
Please add this error code to `docs/en/connector-v2/Error-Quick-Reference-Manual.md`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TaoZex commented on a diff in pull request #4304: [Feature][Connector-V2][GoogleFirestore] Support GoogleFirestore Sink
Posted by "TaoZex (via GitHub)" <gi...@apache.org>.
TaoZex commented on code in PR #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304#discussion_r1155073653
##########
seatunnel-connectors-v2/connector-google-firestore/src/main/java/org/apache/seatunnel/connectors/seatunnel/google/firestore/sink/FirestoreSinkWriter.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.google.firestore.sink;
+
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import org.apache.seatunnel.connectors.seatunnel.common.sink.AbstractSinkWriter;
+import org.apache.seatunnel.connectors.seatunnel.google.firestore.config.FirestoreParameters;
+import org.apache.seatunnel.connectors.seatunnel.google.firestore.exception.FirestoreConnectorErrorCode;
+import org.apache.seatunnel.connectors.seatunnel.google.firestore.exception.FirestoreConnectorException;
+import org.apache.seatunnel.connectors.seatunnel.google.firestore.serialize.DefaultSeaTunnelRowSerializer;
+import org.apache.seatunnel.connectors.seatunnel.google.firestore.serialize.SeaTunnelRowSerializer;
+
+import com.google.auth.oauth2.GoogleCredentials;
+import com.google.cloud.firestore.CollectionReference;
+import com.google.cloud.firestore.Firestore;
+import com.google.cloud.firestore.FirestoreOptions;
+import lombok.extern.slf4j.Slf4j;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.util.Base64;
+
+@Slf4j
+public class FirestoreSinkWriter extends AbstractSinkWriter<SeaTunnelRow, Void> {
+
+ private Firestore firestore;
+
+ private CollectionReference collectionReference;
+
+ private SeaTunnelRowSerializer serializer;
+
+ public FirestoreSinkWriter(SeaTunnelRowType seaTunnelRowType, FirestoreParameters parameters)
+ throws IOException {
+ GoogleCredentials credentials;
+ if (parameters.getCredentials() != null) {
+ byte[] bytes = Base64.getDecoder().decode(parameters.getCredentials());
+ credentials = GoogleCredentials.fromStream(new ByteArrayInputStream(bytes));
+ } else {
+ credentials = GoogleCredentials.getApplicationDefault();
+ }
+ FirestoreOptions firestoreOptions =
+ FirestoreOptions.getDefaultInstance()
+ .toBuilder()
+ .setProjectId(parameters.getProjectId())
+ .setCredentials(credentials)
+ .build();
+ this.firestore = firestoreOptions.getService();
+ this.collectionReference = firestore.collection(parameters.getCollection());
+ this.serializer = new DefaultSeaTunnelRowSerializer(seaTunnelRowType);
+ }
+
+ @Override
+ public void write(SeaTunnelRow seaTunnelRow) throws IOException {
+ collectionReference.add(serializer.serialize(seaTunnelRow));
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (firestore != null) {
+ try {
+ firestore.close();
+ } catch (Exception e) {
+ log.error("Close Firestore client failed.", e);
+ throw new FirestoreConnectorException(
+ FirestoreConnectorErrorCode.CLOSE_CLIENT_FAILED, e);
Review Comment:
```suggestion
throw new FirestoreConnectorException(
FirestoreConnectorErrorCode.CLOSE_CLIENT_FAILED, "Close Firestore client failed.", e);
```
Logging and throwing exception can cause log duplication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lvshaokang commented on pull request #4304: [Feature][Connector-V2][GoogleFirestore] Support GoogleFirestore Sink
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on PR #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304#issuecomment-1465787528
@TaoZex
I didn't find any available docker image, so I don't know if it is still possible to do e2e testing.
The following is a screenshot of the manual testing.
<img width="1251" alt="image" src="https://user-images.githubusercontent.com/29084491/224659619-a005330e-3a04-44f0-86f5-43fd55aaffdc.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TaoZex commented on pull request #4304: [Feature][Connector-V2][GoogleFirestore] Support GoogleFirestore Sink
Posted by "TaoZex (via GitHub)" <gi...@apache.org>.
TaoZex commented on PR #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304#issuecomment-1465796319
LGTM. Thanks for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] liugddx commented on pull request #4304: [Feature][Connector-V2][GoogleFirestore] Support GoogleFirestore Sink
Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on PR #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304#issuecomment-1477372132
please add e2e test case. Refer to https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/coding-guide.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #4304: [Feature][Connector-V2][GoogleFirestore] Support GoogleFirestore Sink
Posted by "TyrantLucifer (via GitHub)" <gi...@apache.org>.
TyrantLucifer commented on PR #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304#issuecomment-1492892581
LGTM, thank you for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on a diff in pull request #4304: [Feature][Connector-V2][GoogleFirestore] Support GoogleFirestore Sink
Posted by "TyrantLucifer (via GitHub)" <gi...@apache.org>.
TyrantLucifer commented on code in PR #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304#discussion_r1134815463
##########
seatunnel-connectors-v2/connector-google-firestore/pom.xml:
##########
@@ -0,0 +1,90 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.seatunnel</groupId>
+ <artifactId>seatunnel-connectors-v2</artifactId>
+ <version>${revision}</version>
+ </parent>
+
+ <artifactId>connector-google-firestore</artifactId>
+ <name>SeaTunnel : Connectors V2 : Google Firestore</name>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.seatunnel</groupId>
+ <artifactId>connector-common</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>com.google.cloud</groupId>
+ <artifactId>google-cloud-firestore</artifactId>
+ <version>3.7.10</version>
Review Comment:
Use properties to manage version number.
##########
seatunnel-connectors-v2/connector-google-firestore/pom.xml:
##########
@@ -0,0 +1,90 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.seatunnel</groupId>
+ <artifactId>seatunnel-connectors-v2</artifactId>
+ <version>${revision}</version>
+ </parent>
+
+ <artifactId>connector-google-firestore</artifactId>
+ <name>SeaTunnel : Connectors V2 : Google Firestore</name>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.seatunnel</groupId>
+ <artifactId>connector-common</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>com.google.cloud</groupId>
+ <artifactId>google-cloud-firestore</artifactId>
+ <version>3.7.10</version>
+ </dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>31.1-android</version>
Review Comment:
The same as above.
##########
seatunnel-connectors-v2/connector-google-firestore/src/main/java/org/apache/seatunnel/connectors/seatunnel/google/firestore/sink/FirestoreSinkWriter.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.google.firestore.sink;
+
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import org.apache.seatunnel.connectors.seatunnel.common.sink.AbstractSinkWriter;
+import org.apache.seatunnel.connectors.seatunnel.google.firestore.config.FirestoreParameters;
+import org.apache.seatunnel.connectors.seatunnel.google.firestore.serialize.DefaultSeaTunnelRowSerializer;
+import org.apache.seatunnel.connectors.seatunnel.google.firestore.serialize.SeaTunnelRowSerializer;
+
+import com.google.auth.oauth2.GoogleCredentials;
+import com.google.cloud.firestore.CollectionReference;
+import com.google.cloud.firestore.Firestore;
+import com.google.cloud.firestore.FirestoreOptions;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.util.Base64;
+
+public class FirestoreSinkWriter extends AbstractSinkWriter<SeaTunnelRow, Void> {
+
+ private Firestore firestore;
+
+ private CollectionReference collectionReference;
+
+ private SeaTunnelRowSerializer serializer;
+
+ public FirestoreSinkWriter(SeaTunnelRowType seaTunnelRowType, FirestoreParameters parameters)
+ throws IOException {
+ GoogleCredentials credentials;
+ if (parameters.getCredentials() != null) {
+ byte[] bytes = Base64.getDecoder().decode(parameters.getCredentials());
+ credentials = GoogleCredentials.fromStream(new ByteArrayInputStream(bytes));
+ } else {
+ credentials = GoogleCredentials.getApplicationDefault();
+ }
+ FirestoreOptions firestoreOptions =
+ FirestoreOptions.getDefaultInstance()
+ .toBuilder()
+ .setProjectId(parameters.getProjectId())
+ .setCredentials(credentials)
+ .build();
+ this.firestore = firestoreOptions.getService();
+ this.collectionReference = firestore.collection(parameters.getCollection());
+ this.serializer = new DefaultSeaTunnelRowSerializer(seaTunnelRowType);
+ }
+
+ @Override
+ public void write(SeaTunnelRow seaTunnelRow) throws IOException {
+ collectionReference.add(serializer.serialize(seaTunnelRow));
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (firestore != null) {
+ try {
+ firestore.close();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
Review Comment:
Unified exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ic4y commented on pull request #4304: [Feature][Connector][GoogleFirestore-Sink] Support GoogleFirestore Sink
Posted by "ic4y (via GitHub)" <gi...@apache.org>.
ic4y commented on PR #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304#issuecomment-1526955506
@lvshaokang Suggest adding the [new version documentation](https://github.com/apache/incubator-seatunnel/issues/4544) for GoogleFirestore Sink in subsequent PR.Or create a related issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lvshaokang commented on pull request #4304: [Feature][Connector-V2][GoogleFirestore] Support GoogleFirestore Sink
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on PR #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304#issuecomment-1469292341
@TyrantLucifer PTAL. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TaoZex commented on pull request #4304: [Feature][Connector-V2][GoogleFirestore] Support GoogleFirestore Sink
Posted by "TaoZex (via GitHub)" <gi...@apache.org>.
TaoZex commented on PR #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304#issuecomment-1465768953
Please provide a screenshot or e2e test to prove that it is available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lvshaokang commented on pull request #4304: [Feature][Connector-V2][GoogleFirestore] Support GoogleFirestore Sink
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on PR #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304#issuecomment-1477383993
> @TaoZex I didn't find any available docker image, so I don't know if it is still possible to do e2e testing. The following is a screenshot of the manual testing. <img alt="image" width="1251" src="https://user-images.githubusercontent.com/29084491/224659619-a005330e-3a04-44f0-86f5-43fd55aaffdc.png">
refer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ic4y merged pull request #4304: [Feature][Connector][GoogleFirestore-Sink] Support GoogleFirestore Sink
Posted by "ic4y (via GitHub)" <gi...@apache.org>.
ic4y merged PR #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ic4y commented on a diff in pull request #4304: [Feature][Connector][GoogleFirestore-Sink] Support GoogleFirestore Sink
Posted by "ic4y (via GitHub)" <gi...@apache.org>.
ic4y commented on code in PR #4304:
URL: https://github.com/apache/incubator-seatunnel/pull/4304#discussion_r1173269264
##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-google-firestore-e2e/src/test/java/org.apache.seatunnel.e2e.connector.google.firestore/GoogleFirestoreIT.java:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.e2e.connector.google.firestore;
+
+import org.apache.seatunnel.connectors.seatunnel.google.firestore.config.FirestoreConfig;
+import org.apache.seatunnel.e2e.common.TestResource;
+import org.apache.seatunnel.e2e.common.TestSuiteBase;
+import org.apache.seatunnel.e2e.common.container.TestContainer;
+import org.apache.seatunnel.e2e.common.util.ContainerUtil;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.TestTemplate;
+import org.testcontainers.containers.Container;
+
+import com.google.api.core.ApiFuture;
+import com.google.auth.oauth2.GoogleCredentials;
+import com.google.cloud.firestore.CollectionReference;
+import com.google.cloud.firestore.Firestore;
+import com.google.cloud.firestore.FirestoreOptions;
+import com.google.cloud.firestore.QueryDocumentSnapshot;
+import com.google.cloud.firestore.QuerySnapshot;
+import com.typesafe.config.Config;
+import com.typesafe.config.ConfigFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.util.Base64;
+import java.util.List;
+
+@Disabled("Disabled because it needs google firestore database to run this test")
+public class GoogleFirestoreIT extends TestSuiteBase implements TestResource {
+
+ private static final String FIRESTORE_CONF_FILE = "/firestore/fake_to_google_firestore.conf";
+
+ private String projectId;
+ private String collection;
+ private String credentials;
+ private Firestore db;
+ private CollectionReference collectionReference;
+
+ @BeforeAll
+ @Override
+ public void startUp() throws Exception {
+ initFirestoreConfig();
+ FirestoreOptions firestoreOptions =
+ FirestoreOptions.getDefaultInstance()
+ .toBuilder()
+ .setProjectId(projectId)
+ .setCredentials(
+ GoogleCredentials.fromStream(
+ new ByteArrayInputStream(
+ Base64.getDecoder().decode(credentials))))
+ .build();
+ this.db = firestoreOptions.getService();
+ this.collectionReference = db.collection(collection);
+ }
+
+ private void initFirestoreConfig() {
+ File file = ContainerUtil.getResourcesFile(FIRESTORE_CONF_FILE);
+ Config config = ConfigFactory.parseFile(file);
+ Config firestoreConfig = config.getConfig("sink").getConfig("GoogleFirestore");
+ this.projectId = firestoreConfig.getString(FirestoreConfig.PROJECT_ID.key());
+ this.collection = firestoreConfig.getString(FirestoreConfig.COLLECTION.key());
+ this.credentials = firestoreConfig.getString(FirestoreConfig.CREDENTIALS.key());
+ }
+
+ @AfterAll
+ @Override
+ public void tearDown() throws Exception {
+ if (db != null) {
+ db.close();
+ }
+ }
+
+ @TestTemplate
+ public void testGoogleFirestore(TestContainer container) throws Exception {
+ Container.ExecResult execResult = container.executeJob(FIRESTORE_CONF_FILE);
+ Assertions.assertEquals(0, execResult.getExitCode());
+
+ List<QueryDocumentSnapshot> documents = readSinkDataset();
+ Assertions.assertTrue(documents.size() > 1);
+ Assertions.assertEquals(15, documents.get(0).getData().size());
Review Comment:
Refer to src/test/java/org/apache/seatunnel/e2e/connector/starrocks/StarRocksCDCSinkIT.java, it is necessary to check the value of each field in each row
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org