You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/10/19 16:30:37 UTC
[GitHub] [incubator-seatunnel] harveyyue opened a new pull request, #3146: [Improve][Connector-V2][ElasticSearch] Add index_id option to specify the primary keys for index
harveyyue opened a new pull request, #3146:
URL: https://github.com/apache/incubator-seatunnel/pull/3146
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
## Purpose of this pull request
<!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
## Check list
* [ ] Code changed are covered with tests, or it does not need tests for reason:
* [ ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] iture123 commented on pull request #3146: [Improve][Connector-V2][ElasticSearch] Add index_id option to specify the primary keys for index
Posted by GitBox <gi...@apache.org>.
iture123 commented on PR #3146:
URL: https://github.com/apache/incubator-seatunnel/pull/3146#issuecomment-1310013356
Jackson conflict has been fixed in latest code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #3146: [Improve][Connector-V2][ElasticSearch] Add index_id option to specify the primary keys for index
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on PR #3146:
URL: https://github.com/apache/incubator-seatunnel/pull/3146#issuecomment-1312450152
cc @iture123 PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] harveyyue commented on a diff in pull request #3146: [Improve][Connector-V2][ElasticSearch] Add index_id option to specify the primary keys for index
Posted by GitBox <gi...@apache.org>.
harveyyue commented on code in PR #3146:
URL: https://github.com/apache/incubator-seatunnel/pull/3146#discussion_r1001648806
##########
seatunnel-connectors-v2/connector-elasticsearch/src/main/java/org/apache/seatunnel/connectors/seatunnel/elasticsearch/serialize/ElasticsearchRowSerializer.java:
##########
@@ -31,41 +38,56 @@
import com.fasterxml.jackson.databind.SerializationFeature;
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
+import java.util.ArrayList;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
+import java.util.stream.Collectors;
/**
* use in elasticsearch version >= 7.*
*/
public class ElasticsearchRowSerializer implements SeaTunnelRowSerializer {
- private final SeaTunnelRowType seaTunnelRowType;
private final ObjectMapper objectMapper = new ObjectMapper().registerModule(new JavaTimeModule())
- .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
+ .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
+ private final SeaTunnelRowType seaTunnelRowType;
+ private final IndexInfo indexInfo;
private final IndexSerializer indexSerializer;
-
private final IndexTypeSerializer indexTypeSerializer;
+ private boolean hasKeys;
public ElasticsearchRowSerializer(ElasticsearchVersion elasticsearchVersion, IndexInfo indexInfo, SeaTunnelRowType seaTunnelRowType) {
this.indexTypeSerializer = IndexTypeSerializerFactory.getIndexTypeSerializer(elasticsearchVersion, indexInfo.getType());
this.indexSerializer = IndexSerializerFactory.getIndexSerializer(indexInfo.getIndex(), seaTunnelRowType);
this.seaTunnelRowType = seaTunnelRowType;
+ this.indexInfo = indexInfo;
+ this.hasKeys = indexInfo.getIds() != null && indexInfo.getIds().size() > 0 ? true : false;
Review Comment:
fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] harveyyue commented on pull request #3146: [Improve][Connector-V2][ElasticSearch] Add index_id option to specify the primary keys for index
Posted by GitBox <gi...@apache.org>.
harveyyue commented on PR #3146:
URL: https://github.com/apache/incubator-seatunnel/pull/3146#issuecomment-1312441576
Have fixed the code conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer closed pull request #3146: [Improve][Connector-V2][ElasticSearch] Add index_id option to specify the primary keys for index
Posted by "TyrantLucifer (via GitHub)" <gi...@apache.org>.
TyrantLucifer closed pull request #3146: [Improve][Connector-V2][ElasticSearch] Add index_id option to specify the primary keys for index
URL: https://github.com/apache/incubator-seatunnel/pull/3146
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] harveyyue commented on a diff in pull request #3146: [Improve][Connector-V2][ElasticSearch] Add index_id option to specify the primary keys for index
Posted by GitBox <gi...@apache.org>.
harveyyue commented on code in PR #3146:
URL: https://github.com/apache/incubator-seatunnel/pull/3146#discussion_r1001642878
##########
seatunnel-connectors-v2/connector-elasticsearch/src/main/java/org/apache/seatunnel/connectors/seatunnel/elasticsearch/dto/IndexInfo.java:
##########
@@ -19,19 +19,25 @@
import org.apache.seatunnel.connectors.seatunnel.elasticsearch.config.SinkConfig;
+import java.util.List;
+
/**
* index config by seatunnel
*/
public class IndexInfo {
private String index;
private String type;
+ private List<String> ids;
Review Comment:
It should be a null value if not setting index_id.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on pull request #3146: [Improve][Connector-V2][ElasticSearch] Add index_id option to specify the primary keys for index
Posted by GitBox <gi...@apache.org>.
hailin0 commented on PR #3146:
URL: https://github.com/apache/incubator-seatunnel/pull/3146#issuecomment-1343798567
https://github.com/apache/incubator-seatunnel/pull/3673
Please merge this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] harveyyue commented on pull request #3146: [Improve][Connector-V2][ElasticSearch] Add index_id option to specify the primary keys for index
Posted by GitBox <gi...@apache.org>.
harveyyue commented on PR #3146:
URL: https://github.com/apache/incubator-seatunnel/pull/3146#issuecomment-1286787232
Currently, I disabled this e2e module because the spark container is running without any response, but the flink is work fine.
Flink log:
`==================== ExecuteConfigFile: /tmp/fake-to-elasticsearch.conf STDOUT end ====================
22/10/21 18:01:36 WARN RestClient: request [POST http://localhost:65164/seatunnel_test_e2e/_search] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security."]
22/10/21 18:01:36 INFO 11_hadoop27:jobmanager]: STDOUT: 2022-10-21 10:01:36,845 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@172.18.0.4:41155] has failed, address is now gated for [50] ms. Reason: [Disassociated]
22/10/21 18:01:36 INFO 11_hadoop27:jobmanager]: STDOUT: 2022-10-21 10:01:36,846 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@172.18.0.4:37045] has failed, address is now gated for [50] ms. Reason: [Disassociated]
22/10/21 18:01:37 INFO TestCaseInvocationContextProvider: The TestContainer[Flink:1.13.6] is closed.
22/10/21 18:01:37 INFO TestLoggerExtension: `
Spark log:
`22/10/21 18:01:38 INFO 6]: Container bitnami/spark:2.4.6 is starting: 3720681794bea6b4daf5db1cfcf66dc576bd1ec6e869a41fe0f75f506dcb690b
22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.73
22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.73 Welcome to the Bitnami spark container
22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.74 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-spark
22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.74 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-spark/issues
22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.74
22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.74 INFO ==> ** Starting Spark setup **
22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.83 INFO ==> Generating Spark configuration file...
22/10/21 18:01:38 INFO 6]: STDOUT:
22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.84 INFO ==> ** Spark setup finished! **
22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.86 INFO ==> ** Starting Spark in master mode **
22/10/21 18:01:38 INFO 6]: STDOUT: starting org.apache.spark.deploy.master.Master, logging to /opt/bitnami/spark/logs/spark--org.apache.spark.deploy.master.Master-1-3720681794be.out
22/10/21 18:01:39 INFO 6]: STDERR: Spark Command: /opt/bitnami/java/bin/java -cp /opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host 3720681794be --port 7077 --webui-port 8080
22/10/21 18:01:39 INFO 6]: STDERR: ========================================
22/10/21 18:01:40 INFO 6]: STDERR: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
22/10/21 18:01:40 INFO 6]: STDERR: 22/10/21 10:01:40 INFO Master: Started daemon with process name: 56@3720681794be
22/10/21 18:01:40 INFO 6]: STDERR: 22/10/21 10:01:40 INFO SignalUtils: Registered signal handler for TERM
22/10/21 18:01:40 INFO 6]: STDERR: 22/10/21 10:01:40 INFO SignalUtils: Registered signal handler for HUP
22/10/21 18:01:40 INFO 6]: STDERR: 22/10/21 10:01:40 INFO SignalUtils: Registered signal handler for INT
22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO SecurityManager: Changing view acls to: spark
22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO SecurityManager: Changing modify acls to: spark
22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO SecurityManager: Changing view acls groups to:
22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO SecurityManager: Changing modify acls groups to:
22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); groups with view permissions: Set(); users with modify permissions: Set(spark); groups with modify permissions: Set()
22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO Master: Starting Spark master at spark://3720681794be:7077
22/10/21 18:01:41 INFO 6]: Container bitnami/spark:2.4.6 started in PT4.363S
22/10/21 18:01:41 INFO TestCaseInvocationContextProvider: The TestContainer[Spark:2.4.3] is running.
22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO Master: Running Spark version 2.4.6
22/10/21 18:01:42 INFO 6]: STDERR: 22/10/21 10:01:42 INFO Utils: Successfully started service 'MasterUI' on port 8080.
22/10/21 18:01:42 INFO 6]: STDERR: 22/10/21 10:01:42 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://3720681794be:8080
22/10/21 18:01:42 INFO 6]: STDERR: 22/10/21 10:01:42 INFO Master: I have been elected leader! New state: ALIVE`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #3146: [Improve][Connector-V2][ElasticSearch] Add index_id option to specify the primary keys for index
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on PR #3146:
URL: https://github.com/apache/incubator-seatunnel/pull/3146#issuecomment-1290550534
> > Currently, I disabled this e2e module because the spark container is running without any response, but the flink is work fine.
> > Flink log: `==================== ExecuteConfigFile: /tmp/fake-to-elasticsearch.conf STDOUT end ==================== 22/10/21 18:01:36 WARN RestClient: request [POST http://localhost:65164/seatunnel_test_e2e/_search] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security."] 22/10/21 18:01:36 INFO 11_hadoop27:jobmanager]: STDOUT: 2022-10-21 10:01:36,845 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@172.18.0.4:41155] has failed, address is now gated for [50] ms. Reason: [Disassociated] 22/10/21 18:01:36 INFO 11_hadoop27:jobmanager]: STDOUT: 2022-10-21 10:01:36,846 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp:/
/flink@172.18.0.4:37045] has failed, address is now gated for [50] ms. Reason: [Disassociated] 22/10/21 18:01:37 INFO TestCaseInvocationContextProvider: The TestContainer[Flink:1.13.6] is closed. 22/10/21 18:01:37 INFO TestLoggerExtension: `
> > Spark log: `22/10/21 18:01:38 INFO 6]: Container bitnami/spark:2.4.6 is starting: 3720681794bea6b4daf5db1cfcf66dc576bd1ec6e869a41fe0f75f506dcb690b 22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.73 22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.73 Welcome to the Bitnami spark container 22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.74 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-spark 22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.74 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-spark/issues 22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.74 22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.74 INFO ==> ** Starting Spark setup ** 22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.83 INFO ==> Generating Spark configuration file... 22/10/21 18:01:38 INFO 6]: STDOUT: 22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.84 INFO ==> ** Spark setup finished! ** 22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.86 INFO ==> ** Starting Spark in master mode **
22/10/21 18:01:38 INFO 6]: STDOUT: starting org.apache.spark.deploy.master.Master, logging to /opt/bitnami/spark/logs/spark--org.apache.spark.deploy.master.Master-1-3720681794be.out 22/10/21 18:01:39 INFO 6]: STDERR: Spark Command: /opt/bitnami/java/bin/java -cp /opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host 3720681794be --port 7077 --webui-port 8080 22/10/21 18:01:39 INFO 6]: STDERR: ======================================== 22/10/21 18:01:40 INFO 6]: STDERR: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 22/10/21 18:01:40 INFO 6]: STDERR: 22/10/21 10:01:40 INFO Master: Started daemon with process name: 56@3720681794be 22/10/21 18:01:40 INFO 6]: STDERR: 22/10/21 10:01:40 INFO SignalUtils: Registered signal handler for TERM 22/10/21 18:01:40 INFO 6]: STDERR: 22/10/21 10:01:40 INFO SignalUtils: Registered signal handler for HUP 22/10/21 18:01:40 INFO 6]: STDERR: 22/10/21 10:01:40 INFO SignalUtils
: Registered signal handler for INT 22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO SecurityManager: Changing view acls to: spark 22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO SecurityManager: Changing modify acls to: spark 22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO SecurityManager: Changing view acls groups to: 22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO SecurityManager: Changing modify acls groups to: 22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); groups with view permissions: Set(); users with modify permissions: Set(spark); groups with modify permissions: Set() 22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO Utils: Su
ccessfully started service 'sparkMaster' on port 7077. 22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO Master: Starting Spark master at spark://3720681794be:7077 22/10/21 18:01:41 INFO 6]: Container bitnami/spark:2.4.6 started in PT4.363S 22/10/21 18:01:41 INFO TestCaseInvocationContextProvider: The TestContainer[Spark:2.4.3] is running. 22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO Master: Running Spark version 2.4.6 22/10/21 18:01:42 INFO 6]: STDERR: 22/10/21 10:01:42 INFO Utils: Successfully started service 'MasterUI' on port 8080. 22/10/21 18:01:42 INFO 6]: STDERR: 22/10/21 10:01:42 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://3720681794be:8080 22/10/21 18:01:42 INFO 6]: STDERR: 22/10/21 10:01:42 INFO Master: I have been elected leader! New state: ALIVE`
>
> @Hisoka-X @TyrantLucifer Do you have any suggestions on this problem?
May be jackson version conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on pull request #3146: [Improve][Connector-V2][ElasticSearch] Add index_id option to specify the primary keys for index
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3146:
URL: https://github.com/apache/incubator-seatunnel/pull/3146#issuecomment-1501005582
This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on a diff in pull request #3146: [Improve][Connector-V2][ElasticSearch] Add index_id option to specify the primary keys for index
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on code in PR #3146:
URL: https://github.com/apache/incubator-seatunnel/pull/3146#discussion_r1001405329
##########
seatunnel-connectors-v2/connector-elasticsearch/src/main/java/org/apache/seatunnel/connectors/seatunnel/elasticsearch/dto/IndexInfo.java:
##########
@@ -19,19 +19,25 @@
import org.apache.seatunnel.connectors.seatunnel.elasticsearch.config.SinkConfig;
+import java.util.List;
+
/**
* index config by seatunnel
*/
public class IndexInfo {
private String index;
private String type;
+ private List<String> ids;
Review Comment:
suggest make ids final if it will never been update.
##########
seatunnel-connectors-v2/connector-elasticsearch/src/main/java/org/apache/seatunnel/connectors/seatunnel/elasticsearch/serialize/ElasticsearchRowSerializer.java:
##########
@@ -31,41 +38,56 @@
import com.fasterxml.jackson.databind.SerializationFeature;
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
+import java.util.ArrayList;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
+import java.util.stream.Collectors;
/**
* use in elasticsearch version >= 7.*
*/
public class ElasticsearchRowSerializer implements SeaTunnelRowSerializer {
- private final SeaTunnelRowType seaTunnelRowType;
private final ObjectMapper objectMapper = new ObjectMapper().registerModule(new JavaTimeModule())
- .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
+ .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
+ private final SeaTunnelRowType seaTunnelRowType;
+ private final IndexInfo indexInfo;
private final IndexSerializer indexSerializer;
-
private final IndexTypeSerializer indexTypeSerializer;
+ private boolean hasKeys;
public ElasticsearchRowSerializer(ElasticsearchVersion elasticsearchVersion, IndexInfo indexInfo, SeaTunnelRowType seaTunnelRowType) {
this.indexTypeSerializer = IndexTypeSerializerFactory.getIndexTypeSerializer(elasticsearchVersion, indexInfo.getType());
this.indexSerializer = IndexSerializerFactory.getIndexSerializer(indexInfo.getIndex(), seaTunnelRowType);
this.seaTunnelRowType = seaTunnelRowType;
+ this.indexInfo = indexInfo;
+ this.hasKeys = indexInfo.getIds() != null && indexInfo.getIds().size() > 0 ? true : false;
Review Comment:
suggest use `CollectionUtils.isEmpty(indexInfo)`
##########
seatunnel-connectors-v2/connector-elasticsearch/src/main/java/org/apache/seatunnel/connectors/seatunnel/elasticsearch/serialize/ElasticsearchRowSerializer.java:
##########
@@ -81,4 +103,28 @@ public String serializeRow(SeaTunnelRow row){
return sb.toString();
}
+
+ private String convertKey(SeaTunnelDataType<?> dataType, Object value) {
+ if (dataType instanceof BasicType
+ || dataType instanceof DecimalType
+ || dataType instanceof PrimitiveByteArrayType
+ || dataType instanceof LocalTimeType) {
+ return String.valueOf(value);
+ } else if (dataType instanceof MapType) {
Review Comment:
You already `return` in the if case, So `else if` is redundant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on pull request #3146: [Improve][Connector-V2][ElasticSearch] Add index_id option to specify the primary keys for index
Posted by GitBox <gi...@apache.org>.
hailin0 commented on PR #3146:
URL: https://github.com/apache/incubator-seatunnel/pull/3146#issuecomment-1285474459
Can you add e2e test for this feature?
reference
https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on pull request #3146: [Improve][Connector-V2][ElasticSearch] Add index_id option to specify the primary keys for index
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on PR #3146:
URL: https://github.com/apache/incubator-seatunnel/pull/3146#issuecomment-1289874453
> Currently, I disabled this e2e module because the spark container is running without any response, but the flink is work fine.
>
> Flink log: `==================== ExecuteConfigFile: /tmp/fake-to-elasticsearch.conf STDOUT end ==================== 22/10/21 18:01:36 WARN RestClient: request [POST http://localhost:65164/seatunnel_test_e2e/_search] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security."] 22/10/21 18:01:36 INFO 11_hadoop27:jobmanager]: STDOUT: 2022-10-21 10:01:36,845 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@172.18.0.4:41155] has failed, address is now gated for [50] ms. Reason: [Disassociated] 22/10/21 18:01:36 INFO 11_hadoop27:jobmanager]: STDOUT: 2022-10-21 10:01:36,846 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://f
link@172.18.0.4:37045] has failed, address is now gated for [50] ms. Reason: [Disassociated] 22/10/21 18:01:37 INFO TestCaseInvocationContextProvider: The TestContainer[Flink:1.13.6] is closed. 22/10/21 18:01:37 INFO TestLoggerExtension: `
>
> Spark log: `22/10/21 18:01:38 INFO 6]: Container bitnami/spark:2.4.6 is starting: 3720681794bea6b4daf5db1cfcf66dc576bd1ec6e869a41fe0f75f506dcb690b 22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.73 22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.73 Welcome to the Bitnami spark container 22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.74 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-spark 22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.74 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-spark/issues 22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.74 22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.74 INFO ==> ** Starting Spark setup ** 22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.83 INFO ==> Generating Spark configuration file... 22/10/21 18:01:38 INFO 6]: STDOUT: 22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.84 INFO ==> ** Spark setup finished! ** 22/10/21 18:01:38 INFO 6]: STDERR: 10:01:38.86 INFO ==> ** Starting Spark in master mode ** 2
2/10/21 18:01:38 INFO 6]: STDOUT: starting org.apache.spark.deploy.master.Master, logging to /opt/bitnami/spark/logs/spark--org.apache.spark.deploy.master.Master-1-3720681794be.out 22/10/21 18:01:39 INFO 6]: STDERR: Spark Command: /opt/bitnami/java/bin/java -cp /opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host 3720681794be --port 7077 --webui-port 8080 22/10/21 18:01:39 INFO 6]: STDERR: ======================================== 22/10/21 18:01:40 INFO 6]: STDERR: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 22/10/21 18:01:40 INFO 6]: STDERR: 22/10/21 10:01:40 INFO Master: Started daemon with process name: 56@3720681794be 22/10/21 18:01:40 INFO 6]: STDERR: 22/10/21 10:01:40 INFO SignalUtils: Registered signal handler for TERM 22/10/21 18:01:40 INFO 6]: STDERR: 22/10/21 10:01:40 INFO SignalUtils: Registered signal handler for HUP 22/10/21 18:01:40 INFO 6]: STDERR: 22/10/21 10:01:40 INFO SignalUtils:
Registered signal handler for INT 22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO SecurityManager: Changing view acls to: spark 22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO SecurityManager: Changing modify acls to: spark 22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO SecurityManager: Changing view acls groups to: 22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO SecurityManager: Changing modify acls groups to: 22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); groups with view permissions: Set(); users with modify permissions: Set(spark); groups with modify permissions: Set() 22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO Utils: Succ
essfully started service 'sparkMaster' on port 7077. 22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO Master: Starting Spark master at spark://3720681794be:7077 22/10/21 18:01:41 INFO 6]: Container bitnami/spark:2.4.6 started in PT4.363S 22/10/21 18:01:41 INFO TestCaseInvocationContextProvider: The TestContainer[Spark:2.4.3] is running. 22/10/21 18:01:41 INFO 6]: STDERR: 22/10/21 10:01:41 INFO Master: Running Spark version 2.4.6 22/10/21 18:01:42 INFO 6]: STDERR: 22/10/21 10:01:42 INFO Utils: Successfully started service 'MasterUI' on port 8080. 22/10/21 18:01:42 INFO 6]: STDERR: 22/10/21 10:01:42 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://3720681794be:8080 22/10/21 18:01:42 INFO 6]: STDERR: 22/10/21 10:01:42 INFO Master: I have been elected leader! New state: ALIVE`
@Hisoka-X @TyrantLucifer Do you have any suggestions on this problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org