You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/04/15 09:14:18 UTC
[GitHub] [incubator-inlong] vernedeng opened a new pull request, #3733: [INLONG-3674][Manager] DataFlowInfo initialization support plugin
vernedeng opened a new pull request, #3733:
URL: https://github.com/apache/incubator-inlong/pull/3733
### Title Name: [INLONG-3674][Manager] DataFlowInfo initialization support plugin
Fixes #3674
### Motivation
*Explain here the context, and why you're making that change. What is the problem you're trying to solve?*
### Modifications
Use CustomInfoFactory to create different component plugin
including class loading and type checking
### Verifying this change
*(Please pick either of the following options)*
- [ ] This change is a trivial rework/code cleanup without any test coverage.
- [ ] This change is already covered by existing tests, such as:
*(please describe tests)*
- [ ] This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] luchunliang commented on a diff in pull request #3733: [INLONG-3674][Manager] DataFlowInfo initialization support plugin
Posted by GitBox <gi...@apache.org>.
luchunliang commented on code in PR #3733:
URL: https://github.com/apache/incubator-inlong/pull/3733#discussion_r851171219
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/thirdparty/sort/util/SinkInfoUtils.java:
##########
@@ -56,12 +59,20 @@ public class SinkInfoUtils {
private static final String DATA_FORMAT = "yyyyMMddHH";
private static final String TIME_FORMAT = "HHmmss";
private static final String DATA_TIME_FORMAT = "yyyyMMddHHmmss";
+ private static final String KEY_TYPE_CUSTOM_SINK_INFO_GENERATOR = "CustomSinkInfoGenerator.type";
/**
* Create sink info for DataFlowInfo.
*/
public static SinkInfo createSinkInfo(SourceResponse sourceResponse, SinkResponse sinkResponse,
- List<FieldInfo> sinkFields) {
+ List<FieldInfo> sinkFields, Map<String, Object> properties) {
+
+ String loadType = String.valueOf(properties.get(KEY_TYPE_CUSTOM_SINK_INFO_GENERATOR));
+ if (!("null".equalsIgnoreCase(loadType))) {
Review Comment:
为什么不直接判断null，要String.valueOf转成"null"
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/thirdparty/sort/util/SourceInfoUtils.java:
##########
@@ -62,10 +69,19 @@ public static boolean isBinlogAllMigration(SourceResponse sourceResponse) {
*/
public static SourceInfo createSourceInfo(PulsarClusterInfo pulsarCluster, String masterAddress,
ClusterBean clusterBean, InlongGroupInfo groupInfo, InlongStreamInfo streamInfo,
- SourceResponse sourceResponse, List<FieldInfo> sourceFields) {
+ SourceResponse sourceResponse, List<FieldInfo> sourceFields, Map<String, Object> properties) {
+
+ String loadType = String.valueOf(properties.get(KEY_TYPE_CUSTOM_SOURCE_INFO_GENERATOR));
+ if (!("null".equalsIgnoreCase(loadType))) {
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] healchow commented on pull request #3733: [INLONG-3674][Manager] DataFlowInfo initialization support plugin
Posted by GitBox <gi...@apache.org>.
healchow commented on PR #3733:
URL: https://github.com/apache/incubator-inlong/pull/3733#issuecomment-1100137974
The plugin should move to `manager-plugins` module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] healchow commented on pull request #3733: [INLONG-3674][Manager] DataFlowInfo initialization support plugin
Posted by GitBox <gi...@apache.org>.
healchow commented on PR #3733:
URL: https://github.com/apache/inlong/pull/3733#issuecomment-1183089235
The plugin function was already added in #4954.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] dockerzhang commented on a diff in pull request #3733: [INLONG-3674][Manager] DataFlowInfo initialization support plugin
Posted by GitBox <gi...@apache.org>.
dockerzhang commented on code in PR #3733:
URL: https://github.com/apache/incubator-inlong/pull/3733#discussion_r857565084
##########
inlong-manager/manager-web/src/main/resources/application.properties:
##########
@@ -55,8 +55,4 @@ common.http-client.connectionRequestTimeout=3000
# Configure auth plugin
inlong.auth.type=default
# Configure dataflow generator plugin
-inlong.sort.serialinfo.type=default
-inlong.sort.transinfo.type=default
-inlong.sort.sinkinfo.type=default
-inlong.sort.fieldinfo.type=default
-inlong.sort.sourceinfo.type=default
\ No newline at end of file
+inlong.sort.type=default
Review Comment:
# Configure the plugin to generate data stream metadata
inlong.sort.metadata.type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] healchow closed pull request #3733: [INLONG-3674][Manager] DataFlowInfo initialization support plugin
Posted by GitBox <gi...@apache.org>.
healchow closed pull request #3733: [INLONG-3674][Manager] DataFlowInfo initialization support plugin
URL: https://github.com/apache/inlong/pull/3733
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] vernedeng commented on a diff in pull request #3733: [INLONG-3674][Manager] DataFlowInfo initialization support plugin
Posted by GitBox <gi...@apache.org>.
vernedeng commented on code in PR #3733:
URL: https://github.com/apache/incubator-inlong/pull/3733#discussion_r851175183
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/thirdparty/sort/util/SinkInfoUtils.java:
##########
@@ -56,12 +59,20 @@ public class SinkInfoUtils {
private static final String DATA_FORMAT = "yyyyMMddHH";
private static final String TIME_FORMAT = "HHmmss";
private static final String DATA_TIME_FORMAT = "yyyyMMddHHmmss";
+ private static final String KEY_TYPE_CUSTOM_SINK_INFO_GENERATOR = "CustomSinkInfoGenerator.type";
/**
* Create sink info for DataFlowInfo.
*/
public static SinkInfo createSinkInfo(SourceResponse sourceResponse, SinkResponse sinkResponse,
- List<FieldInfo> sinkFields) {
+ List<FieldInfo> sinkFields, Map<String, Object> properties) {
+
+ String loadType = String.valueOf(properties.get(KEY_TYPE_CUSTOM_SINK_INFO_GENERATOR));
+ if (!("null".equalsIgnoreCase(loadType))) {
Review Comment:
thx, it will reduce one _String_ initialization in the case that there is no custom plugins
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] dockerzhang commented on a diff in pull request #3733: [INLONG-3674][Manager] DataFlowInfo initialization support plugin
Posted by GitBox <gi...@apache.org>.
dockerzhang commented on code in PR #3733:
URL: https://github.com/apache/incubator-inlong/pull/3733#discussion_r857565084
##########
inlong-manager/manager-web/src/main/resources/application.properties:
##########
@@ -55,8 +55,4 @@ common.http-client.connectionRequestTimeout=3000
# Configure auth plugin
inlong.auth.type=default
# Configure dataflow generator plugin
-inlong.sort.serialinfo.type=default
-inlong.sort.transinfo.type=default
-inlong.sort.sinkinfo.type=default
-inlong.sort.fieldinfo.type=default
-inlong.sort.sourceinfo.type=default
\ No newline at end of file
+inlong.sort.type=default
Review Comment:
\# Configure the plugin to generate data stream metadata
inlong.sort.metadata.type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] luchunliang commented on a diff in pull request #3733: [INLONG-3674][Manager] DataFlowInfo initialization support plugin
Posted by GitBox <gi...@apache.org>.
luchunliang commented on code in PR #3733:
URL: https://github.com/apache/incubator-inlong/pull/3733#discussion_r851171219
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/thirdparty/sort/util/SinkInfoUtils.java:
##########
@@ -56,12 +59,20 @@ public class SinkInfoUtils {
private static final String DATA_FORMAT = "yyyyMMddHH";
private static final String TIME_FORMAT = "HHmmss";
private static final String DATA_TIME_FORMAT = "yyyyMMddHHmmss";
+ private static final String KEY_TYPE_CUSTOM_SINK_INFO_GENERATOR = "CustomSinkInfoGenerator.type";
/**
* Create sink info for DataFlowInfo.
*/
public static SinkInfo createSinkInfo(SourceResponse sourceResponse, SinkResponse sinkResponse,
- List<FieldInfo> sinkFields) {
+ List<FieldInfo> sinkFields, Map<String, Object> properties) {
+
+ String loadType = String.valueOf(properties.get(KEY_TYPE_CUSTOM_SINK_INFO_GENERATOR));
+ if (!("null".equalsIgnoreCase(loadType))) {
Review Comment:
You can check NotNull at first. For example:
String loadType = properties.get(KEY_TYPE_CUSTOM_SINK_INFO_GENERATOR);
if (loadType != null) {
CustomSinkInfoGenerator generator = CustomInfoGeneratorFactory.createCustomSinkInfoGenerator(String.valueOf(loadType));
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] vernedeng commented on a diff in pull request #3733: [INLONG-3674][Manager] DataFlowInfo initialization support plugin
Posted by GitBox <gi...@apache.org>.
vernedeng commented on code in PR #3733:
URL: https://github.com/apache/incubator-inlong/pull/3733#discussion_r851175183
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/thirdparty/sort/util/SinkInfoUtils.java:
##########
@@ -56,12 +59,20 @@ public class SinkInfoUtils {
private static final String DATA_FORMAT = "yyyyMMddHH";
private static final String TIME_FORMAT = "HHmmss";
private static final String DATA_TIME_FORMAT = "yyyyMMddHHmmss";
+ private static final String KEY_TYPE_CUSTOM_SINK_INFO_GENERATOR = "CustomSinkInfoGenerator.type";
/**
* Create sink info for DataFlowInfo.
*/
public static SinkInfo createSinkInfo(SourceResponse sourceResponse, SinkResponse sinkResponse,
- List<FieldInfo> sinkFields) {
+ List<FieldInfo> sinkFields, Map<String, Object> properties) {
+
+ String loadType = String.valueOf(properties.get(KEY_TYPE_CUSTOM_SINK_INFO_GENERATOR));
+ if (!("null".equalsIgnoreCase(loadType))) {
Review Comment:
thx, it will reduce one _String_ initialization in the case that there is no custom plugin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] vernedeng commented on a diff in pull request #3733: [INLONG-3674][Manager] DataFlowInfo initialization support plugin
Posted by GitBox <gi...@apache.org>.
vernedeng commented on code in PR #3733:
URL: https://github.com/apache/incubator-inlong/pull/3733#discussion_r851175183
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/thirdparty/sort/util/SinkInfoUtils.java:
##########
@@ -56,12 +59,20 @@ public class SinkInfoUtils {
private static final String DATA_FORMAT = "yyyyMMddHH";
private static final String TIME_FORMAT = "HHmmss";
private static final String DATA_TIME_FORMAT = "yyyyMMddHHmmss";
+ private static final String KEY_TYPE_CUSTOM_SINK_INFO_GENERATOR = "CustomSinkInfoGenerator.type";
/**
* Create sink info for DataFlowInfo.
*/
public static SinkInfo createSinkInfo(SourceResponse sourceResponse, SinkResponse sinkResponse,
- List<FieldInfo> sinkFields) {
+ List<FieldInfo> sinkFields, Map<String, Object> properties) {
+
+ String loadType = String.valueOf(properties.get(KEY_TYPE_CUSTOM_SINK_INFO_GENERATOR));
+ if (!("null".equalsIgnoreCase(loadType))) {
Review Comment:
values in _properties_ are _Object_ format, should convert it to String first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org