You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by "haifxu (via GitHub)" <gi...@apache.org> on 2023/04/27 06:52:42 UTC
[GitHub] [inlong] haifxu opened a new pull request, #7935: [INLONG-7934][Manager] Optimize the serializationType to support debezium json
haifxu opened a new pull request, #7935:
URL: https://github.com/apache/inlong/pull/7935
- Fixes #7934
### Motivation
1. We need to use different data serialization types based on different data sources. For files, various formats such as CSV and JSON can be used, but for MySQL, PostgreSQL, etc., only debezium Json can be used, so it is placed in the `serializationType` of the stream source as a fixed value.
### Modifications
1. Set the `serializationType` in `MQSource` according to the data source type.
2. Set fixed value for source using debezium.
### Verifying this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] haifxu commented on a diff in pull request #7935: [INLONG-7934][Manager] Optimize the serializationType to support debezium json
Posted by "haifxu (via GitHub)" <gi...@apache.org>.
haifxu commented on code in PR #7935:
URL: https://github.com/apache/inlong/pull/7935#discussion_r1178742790
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/source/kafka/KafkaSourceOperator.java:
##########
@@ -131,8 +127,13 @@ public Map<String, List<StreamSource>> getSourcesMap(InlongGroupInfo groupInfo,
if (!Objects.equals(streamId, sourceInfo.getInlongStreamId())) {
continue;
}
- if (StringUtils.isEmpty(kafkaSource.getSerializationType()) && StringUtils.isNotEmpty(
- sourceInfo.getSerializationType())) {
+
+ if (sourceInfo.getSourceType().equalsIgnoreCase(SourceType.FILE)) {
Review Comment:
Yes, `AUTO_PUSH` also can customize the data format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] dockerzhang commented on a diff in pull request #7935: [INLONG-7934][Manager] Optimize the serializationType to support debezium json
Posted by "dockerzhang (via GitHub)" <gi...@apache.org>.
dockerzhang commented on code in PR #7935:
URL: https://github.com/apache/inlong/pull/7935#discussion_r1178728624
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/source/kafka/KafkaSourceOperator.java:
##########
@@ -131,8 +127,13 @@ public Map<String, List<StreamSource>> getSourcesMap(InlongGroupInfo groupInfo,
if (!Objects.equals(streamId, sourceInfo.getInlongStreamId())) {
continue;
}
- if (StringUtils.isEmpty(kafkaSource.getSerializationType()) && StringUtils.isNotEmpty(
- sourceInfo.getSerializationType())) {
+
+ if (sourceInfo.getSourceType().equalsIgnoreCase(SourceType.FILE)) {
Review Comment:
Here should including `AUTO_PUSH`?
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/source/tubemq/TubeMQSourceOperator.java:
##########
@@ -110,17 +110,21 @@ public Map<String, List<StreamSource>> getSourcesMap(InlongGroupInfo groupInfo,
tubeMQSource.setTopic(streamInfo.getMqResource());
tubeMQSource.setGroupId(streamId);
tubeMQSource.setMasterRpc(masterRpc);
- if (StringUtils.isNotBlank(streamInfo.getDataType())) {
- String serializationType = DataTypeEnum.forType(streamInfo.getDataType()).getType();
- tubeMQSource.setSerializationType(serializationType);
- }
tubeMQSource.setIgnoreParseError(streamInfo.getIgnoreParseError());
for (StreamSource sourceInfo : streamSources) {
if (!Objects.equals(streamId, sourceInfo.getInlongStreamId())) {
continue;
}
- tubeMQSource.setSerializationType(sourceInfo.getSerializationType());
+
+ if (sourceInfo.getSourceType().equalsIgnoreCase(SourceType.FILE)) {
Review Comment:
ditto
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/source/pulsar/PulsarSourceOperator.java:
##########
@@ -149,10 +145,16 @@ public Map<String, List<StreamSource>> getSourcesMap(InlongGroupInfo groupInfo,
if (!Objects.equal(streamId, sourceInfo.getInlongStreamId())) {
continue;
}
- if (StringUtils.isEmpty(pulsarSource.getSerializationType())
- && StringUtils.isNotEmpty(sourceInfo.getSerializationType())) {
+
+ if (sourceInfo.getSourceType().equalsIgnoreCase(SourceType.FILE)) {
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #7935: [INLONG-7934][Manager] Optimize the serializationType to support debezium json
Posted by "dockerzhang (via GitHub)" <gi...@apache.org>.
dockerzhang merged PR #7935:
URL: https://github.com/apache/inlong/pull/7935
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #7935: [INLONG-7934][Manager] Optimize the serializationType to support debezium json
Posted by "healchow (via GitHub)" <gi...@apache.org>.
healchow commented on code in PR #7935:
URL: https://github.com/apache/inlong/pull/7935#discussion_r1179080013
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/source/tubemq/TubeMQSourceOperator.java:
##########
@@ -110,17 +110,22 @@ public Map<String, List<StreamSource>> getSourcesMap(InlongGroupInfo groupInfo,
tubeMQSource.setTopic(streamInfo.getMqResource());
tubeMQSource.setGroupId(streamId);
tubeMQSource.setMasterRpc(masterRpc);
- if (StringUtils.isNotBlank(streamInfo.getDataType())) {
- String serializationType = DataTypeEnum.forType(streamInfo.getDataType()).getType();
- tubeMQSource.setSerializationType(serializationType);
- }
tubeMQSource.setIgnoreParseError(streamInfo.getIgnoreParseError());
for (StreamSource sourceInfo : streamSources) {
if (!Objects.equals(streamId, sourceInfo.getInlongStreamId())) {
continue;
}
- tubeMQSource.setSerializationType(sourceInfo.getSerializationType());
+
+ if (sourceInfo.getSourceType().equalsIgnoreCase(SourceType.FILE) ||
Review Comment:
Do not use `equalsIgnoreCase`, because in the usage scenarios of other modules (such as query MySQL, and Dashboard passing parameters), lowercase types are invalid, and we need to limit the generation of dirty data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] haifxu commented on a diff in pull request #7935: [INLONG-7934][Manager] Optimize the serializationType to support debezium json
Posted by "haifxu (via GitHub)" <gi...@apache.org>.
haifxu commented on code in PR #7935:
URL: https://github.com/apache/inlong/pull/7935#discussion_r1179893375
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/source/tubemq/TubeMQSourceOperator.java:
##########
@@ -110,17 +110,22 @@ public Map<String, List<StreamSource>> getSourcesMap(InlongGroupInfo groupInfo,
tubeMQSource.setTopic(streamInfo.getMqResource());
tubeMQSource.setGroupId(streamId);
tubeMQSource.setMasterRpc(masterRpc);
- if (StringUtils.isNotBlank(streamInfo.getDataType())) {
- String serializationType = DataTypeEnum.forType(streamInfo.getDataType()).getType();
- tubeMQSource.setSerializationType(serializationType);
- }
tubeMQSource.setIgnoreParseError(streamInfo.getIgnoreParseError());
for (StreamSource sourceInfo : streamSources) {
if (!Objects.equals(streamId, sourceInfo.getInlongStreamId())) {
continue;
}
- tubeMQSource.setSerializationType(sourceInfo.getSerializationType());
+
+ if (sourceInfo.getSourceType().equalsIgnoreCase(SourceType.FILE) ||
+ sourceInfo.getSourceType().equalsIgnoreCase(SourceType.AUTO_PUSH)) {
+ if (StringUtils.isNotBlank(streamInfo.getDataType())) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #7935: [INLONG-7934][Manager] Optimize the serializationType to support debezium json
Posted by "healchow (via GitHub)" <gi...@apache.org>.
healchow commented on code in PR #7935:
URL: https://github.com/apache/inlong/pull/7935#discussion_r1179081867
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/source/tubemq/TubeMQSourceOperator.java:
##########
@@ -110,17 +110,22 @@ public Map<String, List<StreamSource>> getSourcesMap(InlongGroupInfo groupInfo,
tubeMQSource.setTopic(streamInfo.getMqResource());
tubeMQSource.setGroupId(streamId);
tubeMQSource.setMasterRpc(masterRpc);
- if (StringUtils.isNotBlank(streamInfo.getDataType())) {
- String serializationType = DataTypeEnum.forType(streamInfo.getDataType()).getType();
- tubeMQSource.setSerializationType(serializationType);
- }
tubeMQSource.setIgnoreParseError(streamInfo.getIgnoreParseError());
for (StreamSource sourceInfo : streamSources) {
if (!Objects.equals(streamId, sourceInfo.getInlongStreamId())) {
continue;
}
- tubeMQSource.setSerializationType(sourceInfo.getSerializationType());
+
+ if (sourceInfo.getSourceType().equalsIgnoreCase(SourceType.FILE) ||
+ sourceInfo.getSourceType().equalsIgnoreCase(SourceType.AUTO_PUSH)) {
+ if (StringUtils.isNotBlank(streamInfo.getDataType())) {
Review Comment:
And also, can we extract those common codes into `AbstractSourceOperator`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org