You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/11/24 05:27:27 UTC

[GitHub] [inlong] yunqingmoswu opened a new pull request, #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

yunqingmoswu opened a new pull request, #6618:
URL: https://github.com/apache/inlong/pull/6618

   ### Prepare a Pull Request
   *(Change the title refer to the following example)*
   
   Title: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink
   
   *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)*
   
   - Fixes #6617 
   
   ### Motivation
   
   Add common process for dirty data sink and supports log sink.
   This piece is designed as follows:
   1. Add option 'dirty.ignore' to determine whether to ignore dirty data, default is 'false'
   2. If the option 'dirty.ignore' is 'true', it can be determined whether to enable sink for dirty data by option 'dirty.sink.enable'
   3. If the option 'dirty.sink.enable' is 'true', The dirty data can be sink to the target system by option 'sink.dirty.connector'
   4. Only supports 'log' sink for dirty data for now.
   6. Add labels for dirty data by option 'sink.dirty.labels', the labels format is 'key1=value1&key2=value2', it supports variable extraction like '${variable}'. There are two system variables[SYSTEM_TIME|DIRTY_TYPE] are currently supported, and the support of other variables is determined by the connector.
   7. Add option 'sink.dirty.format' to format the real dirty data and labels, only supports [csv|json] for now, default is 'csv'.
   8. Add option 'sink.dirty.log-tag' for log dirty data, it appears only in logs before labels.
   9. Add option 'sink.dirty.field-delimiter' to delimit labels and dirty data, default is ','.
   10. If 'sink.dirty.format' is 'csv', the whole log format will be '[${logTag}] ${labels}${fieldDelimiter}${dirtyData}'.
   
   ### Modifications
   
   1.Add dirty process model
   2.Add abstract for DirtySinkFactory, DirtySink,DirtyData,DirtyType
   3.Add DirtyOptions as dirty common options
   4.Add utils class for dirty sink such as LabelUtils, PatternReplaceUtils, and etc.
   5.Add common option define, such as 'dirty.ignore',dirty.sink.enable', 'sink.dirty.connector', 'sink.dirty.labels', 'sink.dirty.format', 'sink.dirty.log-tag', 'sink.dirty.field-delimiter', and etc.
   
   ### Verifying this change
   
   *(Please pick either of the following options)*
   
   - [ ] This change is a trivial rework/code cleanup without any test coverage.
   
   - [x] This change is already covered by existing tests, such as:
     *(please describe tests)*
   
   - [ ] This change added tests and can be verified as follows:
   
     *(example:)*
     - *Added integration tests for end-to-end deployment with large payloads (10MB)*
     - *Extended integration test for recovery after broker failure*
   
   ### Documentation
   
     - Does this pull request introduce a new feature? (yes / no)
     - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
     - If a feature is not applicable for documentation, explain why?
     - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] thesumery commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
thesumery commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1033077419


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/util/LabelUtils.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.util;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import static org.apache.inlong.sort.base.Constants.DELIMITER;
+import static org.apache.inlong.sort.base.Constants.KEY_VALUE_DELIMITER;
+
+/**
+ * The lable utils class, it is used for parse the lables to a label map
+ */
+public final class LabelUtils {
+
+    private LabelUtils() {
+    }
+
+    /**
+     * Parse the labels to label map
+     *
+     * @param labels The labels format by 'key1=value1&key2=value2...'
+     * @return The label map of labels
+     */
+    public static Map<String, String> parseLabels(String labels) {
+        return parseLabels(labels, new LinkedHashMap<>());
+    }
+
+    /**
+     * Parse the labels to label map
+     *
+     * @param labels The labels format by 'key1=value1&key2=value2...'
+     * @return The label map of labels
+     */
+    public static Map<String, String> parseLabels(String labels, Map<String, String> labelMap) {
+        if (labelMap == null) {
+            labelMap = new LinkedHashMap<>();
+        }
+        if (labels == null || labels.length() == 0) {
+            return labelMap;
+        }
+        String[] labelArray = labels.split(DELIMITER);
+        for (String label : labelArray) {
+            int index = label.indexOf(KEY_VALUE_DELIMITER);
+            if (index < 1 || index == label.length() - 1) {
+                throw new IllegalArgumentException("The format of option 'sink.dirty.labels' must be key=value");

Review Comment:
   LabelUtils is only used for 'dirty.lables'?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] yunqingmoswu commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1032167598


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java:
##########
@@ -177,4 +181,93 @@ public final class Constants {
                     .defaultValue(false)
                     .withDescription("Because spark do not support iceberg data type: `timestamp without time zone` and"
                             + "`time`, so type conversions must be mapped to types supported by spark.");
+
+    //========================================= dirty configuration =========================================
+    public static final String DEFAULT_LABEL_IDENTIFER = "dirty-data";
+    public static final String SINK_DIRTY_PREFIX = "sink.dirty.";
+    public static final String DIRTY_PREFIX = "dirty.";
+    public static final String DIRTY_LABEL_DELIMITER = "-";
+
+    public static final ConfigOption<Boolean> DIRTY_IGNORE =

Review Comment:
   It has stated in the description.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] yunqingmoswu commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1033192551


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/util/PatternReplaceUtils.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.util;
+
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * The pattern replace utils
+ */
+public final class PatternReplaceUtils {
+
+    private static final Pattern REGEX_PATTERN = Pattern.compile("\\$\\{\\s*([\\w.-]+)\\s*}",
+            Pattern.CASE_INSENSITIVE);
+
+    public static String replace(String pattern, Map<String, String> params) {
+        if (pattern == null) {
+            return pattern;
+        }
+        Matcher matcher = REGEX_PATTERN.matcher(pattern);
+        StringBuffer sb = new StringBuffer();
+        while (matcher.find()) {
+            String keyText = matcher.group(1);
+            String replacement = params.getOrDefault(keyText, keyText);
+            matcher.appendReplacement(sb, replacement);

Review Comment:
   > Log if there is placeholder has no defaultValue?
   
   Personally, I don’t think it is necessary to add a log here, the information is already reflected in the connector parameters.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] yunqingmoswu closed pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
yunqingmoswu closed pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink
URL: https://github.com/apache/inlong/pull/6618


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] thesumery commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
thesumery commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1033081321


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/dirty/sink/log/LogDirtySink.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.dirty.sink.log;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.formats.common.TimestampFormat;
+import org.apache.flink.formats.json.JsonOptions.MapNullKeyMode;
+import org.apache.flink.formats.json.RowDataToJsonConverters;
+import org.apache.flink.formats.json.RowDataToJsonConverters.RowDataToJsonConverter;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.inlong.sort.base.dirty.DirtyData;
+import org.apache.inlong.sort.base.dirty.sink.DirtySink;
+import org.apache.inlong.sort.base.dirty.utils.FormatUtils;
+import org.apache.inlong.sort.base.util.LabelUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+import static org.apache.flink.table.data.RowData.createFieldGetter;
+
+/**
+ * Log dirty sink that is used to print log
+ *
+ * @param <T>
+ */
+public class LogDirtySink<T> implements DirtySink<T> {
+
+    private static final long serialVersionUID = 1L;
+
+    private static final Logger LOG = LoggerFactory.getLogger(LogDirtySink.class);
+
+    private final RowData.FieldGetter[] fieldGetters;
+    private final String format;
+    private final String fieldDelimiter;
+    private final DataType physicalRowDataType;
+    private RowDataToJsonConverter converter;
+
+    public LogDirtySink(String format, String fieldDelimiter, DataType physicalRowDataType) {
+        this.format = format;
+        this.fieldDelimiter = fieldDelimiter;
+        this.physicalRowDataType = physicalRowDataType;
+        final LogicalType[] logicalTypes = physicalRowDataType.getChildren()
+                .stream().map(DataType::getLogicalType).toArray(LogicalType[]::new);
+        this.fieldGetters = new RowData.FieldGetter[logicalTypes.length];

Review Comment:
   Here has a problem, `DataType` is fixed at initialize.
   The `DataType` cannot be given a specific value in the whole database integrated scene, so format data into a format is difficult.Maybe the reason that dirty data produced is that schema mismatched. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] Yizhou-Yang commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
Yizhou-Yang commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1032001028


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java:
##########
@@ -177,4 +181,93 @@ public final class Constants {
                     .defaultValue(false)
                     .withDescription("Because spark do not support iceberg data type: `timestamp without time zone` and"
                             + "`time`, so type conversions must be mapped to types supported by spark.");
+
+    //========================================= dirty configuration =========================================
+    public static final String DEFAULT_LABEL_IDENTIFER = "dirty-data";
+    public static final String SINK_DIRTY_PREFIX = "sink.dirty.";
+    public static final String DIRTY_PREFIX = "dirty.";
+    public static final String DIRTY_LABEL_DELIMITER = "-";
+
+    public static final ConfigOption<Boolean> DIRTY_IGNORE =
+            ConfigOptions.key("dirty.ignore")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription("Whether ignore the dirty data, default value is 'false'");
+    public static final ConfigOption<Boolean> DIRTY_SINK_ENABLE =
+            ConfigOptions.key("dirty.sink.enable")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription("Whether supports dirty data sink, default value is 'false'");
+    public static final ConfigOption<String> SINK_DIRTY_CONNECTOR =
+            ConfigOptions.key("sink.dirty.connector")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription("The connector of dirty sink");
+    public static final ConfigOption<String> SINK_DIRTY_FORMAT =
+            ConfigOptions.key("sink.dirty.format")
+                    .stringType()
+                    .defaultValue("csv")

Review Comment:
   for now only supports csv and json?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] yunqingmoswu commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1033243595


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/util/LabelUtils.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.util;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import static org.apache.inlong.sort.base.Constants.DELIMITER;
+import static org.apache.inlong.sort.base.Constants.KEY_VALUE_DELIMITER;
+
+/**
+ * The lable utils class, it is used for parse the lables to a label map
+ */
+public final class LabelUtils {
+
+    private LabelUtils() {
+    }
+
+    /**
+     * Parse the labels to label map
+     *
+     * @param labels The labels format by 'key1=value1&key2=value2...'
+     * @return The label map of labels
+     */
+    public static Map<String, String> parseLabels(String labels) {
+        return parseLabels(labels, new LinkedHashMap<>());
+    }
+
+    /**
+     * Parse the labels to label map
+     *
+     * @param labels The labels format by 'key1=value1&key2=value2...'
+     * @return The label map of labels
+     */
+    public static Map<String, String> parseLabels(String labels, Map<String, String> labelMap) {
+        if (labelMap == null) {
+            labelMap = new LinkedHashMap<>();
+        }
+        if (labels == null || labels.length() == 0) {
+            return labelMap;
+        }
+        String[] labelArray = labels.split(DELIMITER);
+        for (String label : labelArray) {
+            int index = label.indexOf(KEY_VALUE_DELIMITER);
+            if (index < 1 || index == label.length() - 1) {
+                throw new IllegalArgumentException("The format of option 'sink.dirty.labels' must be key=value");

Review Comment:
   thanks, i will deal with it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] yunqingmoswu commented on pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on PR #6618:
URL: https://github.com/apache/inlong/pull/6618#issuecomment-1327211753

   > how can the connector be applied to the existed connector, can you describe it in the description ?
   
   It is necessary to create DirtySink through DirtyFactory in the connector, and then call DirtySink's invoke method where dirty data occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] EMsnap merged pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
EMsnap merged PR #6618:
URL: https://github.com/apache/inlong/pull/6618


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] thesumery commented on pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
thesumery commented on PR #6618:
URL: https://github.com/apache/inlong/pull/6618#issuecomment-1328901882

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] Yizhou-Yang commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
Yizhou-Yang commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1032004363


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/dirty/DirtyOptions.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.dirty;
+
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.api.ValidationException;
+
+import java.io.Serializable;
+import static org.apache.inlong.sort.base.Constants.DIRTY_IGNORE;
+import static org.apache.inlong.sort.base.Constants.DIRTY_SINK_ENABLE;
+import static org.apache.inlong.sort.base.Constants.SINK_DIRTY_CONNECTOR;
+import static org.apache.inlong.sort.base.Constants.SINK_DIRTY_IGNORE_SINK_ERRORS;
+import static org.apache.inlong.sort.base.Constants.SINK_DIRTY_LABELS;
+import static org.apache.inlong.sort.base.Constants.SINK_DIRTY_LOG_TAG;
+
+/**
+ * Dirty common options
+ */
+public class DirtyOptions implements Serializable {
+
+    private static final long serialVersionUID = 1L;
+
+    private final boolean ignoreDirty;
+    private final boolean enableDirtySink;
+    private final boolean ignoreSinkErrors;
+    private final String dirtyConnector;
+    private final String labels;
+    private final String logTag;
+
+    private DirtyOptions(boolean ignoreDirty, boolean enableDirtySink, boolean ignoreSinkErrors,
+            String dirtyConnector, String labels, String logTag) {
+        this.ignoreDirty = ignoreDirty;
+        this.enableDirtySink = enableDirtySink;
+        this.ignoreSinkErrors = ignoreSinkErrors;
+        this.dirtyConnector = dirtyConnector;
+        this.labels = labels;
+        this.logTag = logTag;
+    }
+
+    /**
+     * Get dirty options from {@link ReadableConfig}
+     *
+     * @param config The config
+     * @return Dirty options
+     */
+    public static DirtyOptions fromConfig(ReadableConfig config) {
+        boolean ignoreDirty = config.get(DIRTY_IGNORE);
+        boolean enableDirtySink = config.get(DIRTY_SINK_ENABLE);
+        boolean ignoreSinkError = config.get(SINK_DIRTY_IGNORE_SINK_ERRORS);
+        String dirtyConnector = config.getOptional(SINK_DIRTY_CONNECTOR).orElse(null);
+        String labels = config.getOptional(SINK_DIRTY_LABELS).orElse(null);
+        String logTag = config.get(SINK_DIRTY_LOG_TAG);

Review Comment:
   can there be dirty data with null log tags?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] yunqingmoswu commented on pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on PR #6618:
URL: https://github.com/apache/inlong/pull/6618#issuecomment-1327204946

   > Maybe, add some decription in issue, how to use ?
   
   It has located in pr describe.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] yunqingmoswu commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1032166559


##########
inlong-sort/sort-connectors/base/src/test/java/org/apache/inlong/sort/base/dirty/FormatUtilsTest.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.dirty;
+
+/**
+ * Test for {@link org.apache.inlong.sort.base.dirty.utils.FormatUtils}
+ */
+public class FormatUtilsTest {

Review Comment:
   I will add some test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] yunqingmoswu commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1033194547


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/dirty/sink/log/LogDirtySink.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.dirty.sink.log;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.formats.common.TimestampFormat;
+import org.apache.flink.formats.json.JsonOptions.MapNullKeyMode;
+import org.apache.flink.formats.json.RowDataToJsonConverters;
+import org.apache.flink.formats.json.RowDataToJsonConverters.RowDataToJsonConverter;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.inlong.sort.base.dirty.DirtyData;
+import org.apache.inlong.sort.base.dirty.sink.DirtySink;
+import org.apache.inlong.sort.base.dirty.utils.FormatUtils;
+import org.apache.inlong.sort.base.util.LabelUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+import static org.apache.flink.table.data.RowData.createFieldGetter;
+
+/**
+ * Log dirty sink that is used to print log
+ *
+ * @param <T>
+ */
+public class LogDirtySink<T> implements DirtySink<T> {
+
+    private static final long serialVersionUID = 1L;
+
+    private static final Logger LOG = LoggerFactory.getLogger(LogDirtySink.class);
+
+    private final RowData.FieldGetter[] fieldGetters;
+    private final String format;
+    private final String fieldDelimiter;
+    private final DataType physicalRowDataType;
+    private RowDataToJsonConverter converter;
+
+    public LogDirtySink(String format, String fieldDelimiter, DataType physicalRowDataType) {
+        this.format = format;
+        this.fieldDelimiter = fieldDelimiter;
+        this.physicalRowDataType = physicalRowDataType;
+        final LogicalType[] logicalTypes = physicalRowDataType.getChildren()
+                .stream().map(DataType::getLogicalType).toArray(LogicalType[]::new);
+        this.fieldGetters = new RowData.FieldGetter[logicalTypes.length];

Review Comment:
   > Here has a problem, `DataType` is fixed at initialize. The `DataType` cannot be given a specific value in the whole database integrated scene, so format data into a format is difficult.Maybe the reason that dirty data produced is that schema mismatched.
   
   Here it is only initialized and not actually called. How to use it depends on the connector and the corresponding scene.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] thesumery commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
thesumery commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1033074026


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java:
##########
@@ -177,4 +181,93 @@ public final class Constants {
                     .defaultValue(false)
                     .withDescription("Because spark do not support iceberg data type: `timestamp without time zone` and"
                             + "`time`, so type conversions must be mapped to types supported by spark.");
+
+    //========================================= dirty configuration =========================================
+    public static final String DEFAULT_LABEL_IDENTIFER = "dirty-data";
+    public static final String SINK_DIRTY_PREFIX = "sink.dirty.";
+    public static final String DIRTY_PREFIX = "dirty.";
+    public static final String DIRTY_LABEL_DELIMITER = "-";
+
+    public static final ConfigOption<Boolean> DIRTY_IGNORE =
+            ConfigOptions.key("dirty.ignore")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription("Whether ignore the dirty data, default value is 'false'");
+    public static final ConfigOption<Boolean> DIRTY_SINK_ENABLE =
+            ConfigOptions.key("dirty.sink.enable")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription("Whether supports dirty data sink, default value is 'false'");
+    public static final ConfigOption<String> SINK_DIRTY_CONNECTOR =
+            ConfigOptions.key("sink.dirty.connector")

Review Comment:
   Is there any ambiguity in the description here. 
   'sink.dirty.connector'  here describe how dirty data is written, but maybe described how dirty data used in sink node. 
   In fact it can be used in source node and sink node.
   So maybe 'dirty.sink.xxx' or 'dirty.side-output.xxx' is better ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] gong commented on pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
gong commented on PR #6618:
URL: https://github.com/apache/inlong/pull/6618#issuecomment-1327602909

   > > how can the connector be applied to the existed connector, can you describe it in the description ?
   > 
   > It is necessary to create DirtySink through DirtyFactory in the connector, and then call DirtySink's invoke method where dirty data occurs.
   
   @yunqingmoswu Maybe, add a tool class like `org.apache.flink.table.factories.FactoryUtil`. It can find sub class of `DirtyFactory` to create `dirtySink` object.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] gong commented on pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
gong commented on PR #6618:
URL: https://github.com/apache/inlong/pull/6618#issuecomment-1327183019

   Maybe, add some decription in issue, how to use ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] Yizhou-Yang commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
Yizhou-Yang commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1032008227


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/dirty/sink/log/LogDirtySink.java:
##########
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.dirty.sink.log;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.inlong.sort.base.dirty.DirtyData;
+import org.apache.inlong.sort.base.dirty.sink.DirtySink;
+import org.apache.inlong.sort.base.dirty.utils.FormatUtils;
+import org.apache.inlong.sort.base.util.LabelUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.Map;
+import static org.apache.flink.table.data.RowData.createFieldGetter;
+
+/**
+ * Log dirty sink that is used to print log
+ *
+ * @param <T>
+ */
+public class LogDirtySink<T> implements DirtySink<T> {
+
+    private static final long serialVersionUID = 1L;
+
+    private static final Logger LOG = LoggerFactory.getLogger(LogDirtySink.class);
+
+    private final RowData.FieldGetter[] fieldGetters;
+    private final String[] fieldNames;
+    private final String format;
+    private final String fieldDelimiter;
+
+    public LogDirtySink(String format, String fieldDelimiter,
+            DataType[] fieldDataTypes, String[] fieldNames) {
+        this.format = format;
+        this.fieldDelimiter = fieldDelimiter;
+        this.fieldNames = fieldNames;
+        final LogicalType[] logicalTypes = Arrays.stream(fieldDataTypes)

Review Comment:
   what are the possible "LogicalType"s for a dirty sink? will there be problems around incompatible logical types?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] yunqingmoswu commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1032168449


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/dirty/DirtyOptions.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.dirty;
+
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.api.ValidationException;
+
+import java.io.Serializable;
+import static org.apache.inlong.sort.base.Constants.DIRTY_IGNORE;
+import static org.apache.inlong.sort.base.Constants.DIRTY_SINK_ENABLE;
+import static org.apache.inlong.sort.base.Constants.SINK_DIRTY_CONNECTOR;
+import static org.apache.inlong.sort.base.Constants.SINK_DIRTY_IGNORE_SINK_ERRORS;
+import static org.apache.inlong.sort.base.Constants.SINK_DIRTY_LABELS;
+import static org.apache.inlong.sort.base.Constants.SINK_DIRTY_LOG_TAG;
+
+/**
+ * Dirty common options
+ */
+public class DirtyOptions implements Serializable {
+
+    private static final long serialVersionUID = 1L;
+
+    private final boolean ignoreDirty;
+    private final boolean enableDirtySink;
+    private final boolean ignoreSinkErrors;
+    private final String dirtyConnector;
+    private final String labels;
+    private final String logTag;
+
+    private DirtyOptions(boolean ignoreDirty, boolean enableDirtySink, boolean ignoreSinkErrors,
+            String dirtyConnector, String labels, String logTag) {
+        this.ignoreDirty = ignoreDirty;
+        this.enableDirtySink = enableDirtySink;
+        this.ignoreSinkErrors = ignoreSinkErrors;
+        this.dirtyConnector = dirtyConnector;
+        this.labels = labels;
+        this.logTag = logTag;
+    }
+
+    /**
+     * Get dirty options from {@link ReadableConfig}
+     *
+     * @param config The config
+     * @return Dirty options
+     */
+    public static DirtyOptions fromConfig(ReadableConfig config) {
+        boolean ignoreDirty = config.get(DIRTY_IGNORE);
+        boolean enableDirtySink = config.get(DIRTY_SINK_ENABLE);
+        boolean ignoreSinkError = config.get(SINK_DIRTY_IGNORE_SINK_ERRORS);
+        String dirtyConnector = config.getOptional(SINK_DIRTY_CONNECTOR).orElse(null);
+        String labels = config.getOptional(SINK_DIRTY_LABELS).orElse(null);
+        String logTag = config.get(SINK_DIRTY_LOG_TAG);

Review Comment:
   A log tag must be required here, and there is a default if the user does not set it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] yunqingmoswu commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1032170650


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/dirty/sink/log/LogDirtySink.java:
##########
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.dirty.sink.log;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.inlong.sort.base.dirty.DirtyData;
+import org.apache.inlong.sort.base.dirty.sink.DirtySink;
+import org.apache.inlong.sort.base.dirty.utils.FormatUtils;
+import org.apache.inlong.sort.base.util.LabelUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.Map;
+import static org.apache.flink.table.data.RowData.createFieldGetter;
+
+/**
+ * Log dirty sink that is used to print log
+ *
+ * @param <T>
+ */
+public class LogDirtySink<T> implements DirtySink<T> {
+
+    private static final long serialVersionUID = 1L;
+
+    private static final Logger LOG = LoggerFactory.getLogger(LogDirtySink.class);
+
+    private final RowData.FieldGetter[] fieldGetters;
+    private final String[] fieldNames;
+    private final String format;
+    private final String fieldDelimiter;
+
+    public LogDirtySink(String format, String fieldDelimiter,
+            DataType[] fieldDataTypes, String[] fieldNames) {
+        this.format = format;
+        this.fieldDelimiter = fieldDelimiter;
+        this.fieldNames = fieldNames;
+        final LogicalType[] logicalTypes = Arrays.stream(fieldDataTypes)

Review Comment:
   It is the type of the data specified by the user when defining the task, when the data is also of the same type.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] yunqingmoswu commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1033243291


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java:
##########
@@ -177,4 +181,93 @@ public final class Constants {
                     .defaultValue(false)
                     .withDescription("Because spark do not support iceberg data type: `timestamp without time zone` and"
                             + "`time`, so type conversions must be mapped to types supported by spark.");
+
+    //========================================= dirty configuration =========================================
+    public static final String DEFAULT_LABEL_IDENTIFER = "dirty-data";
+    public static final String SINK_DIRTY_PREFIX = "sink.dirty.";
+    public static final String DIRTY_PREFIX = "dirty.";
+    public static final String DIRTY_LABEL_DELIMITER = "-";
+
+    public static final ConfigOption<Boolean> DIRTY_IGNORE =
+            ConfigOptions.key("dirty.ignore")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription("Whether ignore the dirty data, default value is 'false'");
+    public static final ConfigOption<Boolean> DIRTY_SINK_ENABLE =
+            ConfigOptions.key("dirty.sink.enable")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription("Whether supports dirty data sink, default value is 'false'");
+    public static final ConfigOption<String> SINK_DIRTY_CONNECTOR =
+            ConfigOptions.key("sink.dirty.connector")

Review Comment:
   It is a good idea, i will deal with it latter.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] yunqingmoswu commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1033194331


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/dirty/sink/log/LogDirtySink.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.dirty.sink.log;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.formats.common.TimestampFormat;
+import org.apache.flink.formats.json.JsonOptions.MapNullKeyMode;
+import org.apache.flink.formats.json.RowDataToJsonConverters;
+import org.apache.flink.formats.json.RowDataToJsonConverters.RowDataToJsonConverter;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.inlong.sort.base.dirty.DirtyData;
+import org.apache.inlong.sort.base.dirty.sink.DirtySink;
+import org.apache.inlong.sort.base.dirty.utils.FormatUtils;
+import org.apache.inlong.sort.base.util.LabelUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+import static org.apache.flink.table.data.RowData.createFieldGetter;
+
+/**
+ * Log dirty sink that is used to print log
+ *
+ * @param <T>
+ */
+public class LogDirtySink<T> implements DirtySink<T> {
+
+    private static final long serialVersionUID = 1L;
+
+    private static final Logger LOG = LoggerFactory.getLogger(LogDirtySink.class);
+
+    private final RowData.FieldGetter[] fieldGetters;
+    private final String format;
+    private final String fieldDelimiter;
+    private final DataType physicalRowDataType;
+    private RowDataToJsonConverter converter;
+
+    public LogDirtySink(String format, String fieldDelimiter, DataType physicalRowDataType) {
+        this.format = format;
+        this.fieldDelimiter = fieldDelimiter;
+        this.physicalRowDataType = physicalRowDataType;
+        final LogicalType[] logicalTypes = physicalRowDataType.getChildren()
+                .stream().map(DataType::getLogicalType).toArray(LogicalType[]::new);
+        this.fieldGetters = new RowData.FieldGetter[logicalTypes.length];

Review Comment:
   Here it is only initialized and not actually called. How to use it depends on the connector and the corresponding scene.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] thesumery commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
thesumery commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1033074026


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java:
##########
@@ -177,4 +181,93 @@ public final class Constants {
                     .defaultValue(false)
                     .withDescription("Because spark do not support iceberg data type: `timestamp without time zone` and"
                             + "`time`, so type conversions must be mapped to types supported by spark.");
+
+    //========================================= dirty configuration =========================================
+    public static final String DEFAULT_LABEL_IDENTIFER = "dirty-data";
+    public static final String SINK_DIRTY_PREFIX = "sink.dirty.";
+    public static final String DIRTY_PREFIX = "dirty.";
+    public static final String DIRTY_LABEL_DELIMITER = "-";
+
+    public static final ConfigOption<Boolean> DIRTY_IGNORE =
+            ConfigOptions.key("dirty.ignore")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription("Whether ignore the dirty data, default value is 'false'");
+    public static final ConfigOption<Boolean> DIRTY_SINK_ENABLE =
+            ConfigOptions.key("dirty.sink.enable")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription("Whether supports dirty data sink, default value is 'false'");
+    public static final ConfigOption<String> SINK_DIRTY_CONNECTOR =
+            ConfigOptions.key("sink.dirty.connector")

Review Comment:
   Is there any ambiguity in the description here. 
   'sink.dirty.connector'  here describe how dirty data is written, but maybe described how dirty data used in sink node. 
   In fact it can be used in source node and sink node.
   So maybe 'dirty.sink.xxx' is better?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] EMsnap commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
EMsnap commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1031980912


##########
inlong-sort/sort-connectors/base/src/test/java/org/apache/inlong/sort/base/dirty/FormatUtilsTest.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.dirty;
+
+/**
+ * Test for {@link org.apache.inlong.sort.base.dirty.utils.FormatUtils}
+ */
+public class FormatUtilsTest {

Review Comment:
   Empty test? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] Yizhou-Yang commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
Yizhou-Yang commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1032002285


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java:
##########
@@ -177,4 +181,93 @@ public final class Constants {
                     .defaultValue(false)
                     .withDescription("Because spark do not support iceberg data type: `timestamp without time zone` and"
                             + "`time`, so type conversions must be mapped to types supported by spark.");
+
+    //========================================= dirty configuration =========================================
+    public static final String DEFAULT_LABEL_IDENTIFER = "dirty-data";
+    public static final String SINK_DIRTY_PREFIX = "sink.dirty.";
+    public static final String DIRTY_PREFIX = "dirty.";
+    public static final String DIRTY_LABEL_DELIMITER = "-";
+
+    public static final ConfigOption<Boolean> DIRTY_IGNORE =

Review Comment:
   what is the difference between ignoring dirty data and ignoring sink errors? should sink error only occur when there is dirty data that is not being ignored?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] Yizhou-Yang commented on pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
Yizhou-Yang commented on PR #6618:
URL: https://github.com/apache/inlong/pull/6618#issuecomment-1327023448

   this pr mainly focuses on classifying and giving corresponding labels to the dirty data...
   Is this dirty sink added to integration tasks as one of the sinks? whenever data is dirty during the process, write to the given sink....
   
   I feel that there is much work to do when adapting this dirty sink to individual adaptors and identifying exactly where errors would occur, and what happens if errors occur during read/writes to the dirty sink? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] yunqingmoswu commented on pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on PR #6618:
URL: https://github.com/apache/inlong/pull/6618#issuecomment-1328449119

   > > > how can the connector be applied to the existed connector, can you describe it in the description ?
   > > 
   > > 
   > > It is necessary to create DirtySink through DirtyFactory in the connector, and then call DirtySink's invoke method where dirty data occurs.
   > 
   > @yunqingmoswu Maybe, add a tool class like `org.apache.flink.table.factories.FactoryUtil`. It can find sub class of `DirtyFactory` to create `dirtySink` object.
   Just use it directly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] thesumery commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
thesumery commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1033076509


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/util/PatternReplaceUtils.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.util;
+
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * The pattern replace utils
+ */
+public final class PatternReplaceUtils {
+
+    private static final Pattern REGEX_PATTERN = Pattern.compile("\\$\\{\\s*([\\w.-]+)\\s*}",
+            Pattern.CASE_INSENSITIVE);
+
+    public static String replace(String pattern, Map<String, String> params) {
+        if (pattern == null) {
+            return pattern;
+        }
+        Matcher matcher = REGEX_PATTERN.matcher(pattern);
+        StringBuffer sb = new StringBuffer();
+        while (matcher.find()) {
+            String keyText = matcher.group(1);
+            String replacement = params.getOrDefault(keyText, keyText);
+            matcher.appendReplacement(sb, replacement);

Review Comment:
   Log if there is placeholder has no defaultValue?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] yunqingmoswu commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1033195349


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/util/LabelUtils.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.util;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import static org.apache.inlong.sort.base.Constants.DELIMITER;
+import static org.apache.inlong.sort.base.Constants.KEY_VALUE_DELIMITER;
+
+/**
+ * The lable utils class, it is used for parse the lables to a label map
+ */
+public final class LabelUtils {
+
+    private LabelUtils() {
+    }
+
+    /**
+     * Parse the labels to label map
+     *
+     * @param labels The labels format by 'key1=value1&key2=value2...'
+     * @return The label map of labels
+     */
+    public static Map<String, String> parseLabels(String labels) {
+        return parseLabels(labels, new LinkedHashMap<>());
+    }
+
+    /**
+     * Parse the labels to label map
+     *
+     * @param labels The labels format by 'key1=value1&key2=value2...'
+     * @return The label map of labels
+     */
+    public static Map<String, String> parseLabels(String labels, Map<String, String> labelMap) {
+        if (labelMap == null) {
+            labelMap = new LinkedHashMap<>();
+        }
+        if (labels == null || labels.length() == 0) {
+            return labelMap;
+        }
+        String[] labelArray = labels.split(DELIMITER);
+        for (String label : labelArray) {
+            int index = label.indexOf(KEY_VALUE_DELIMITER);
+            if (index < 1 || index == label.length() - 1) {
+                throw new IllegalArgumentException("The format of option 'sink.dirty.labels' must be key=value");

Review Comment:
   The code here is repeated with 'inlong.metric.labels', I abstracted it, but there is a problem with the error message, I will deal with it



##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/util/LabelUtils.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.util;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import static org.apache.inlong.sort.base.Constants.DELIMITER;
+import static org.apache.inlong.sort.base.Constants.KEY_VALUE_DELIMITER;
+
+/**
+ * The lable utils class, it is used for parse the lables to a label map
+ */
+public final class LabelUtils {
+
+    private LabelUtils() {
+    }
+
+    /**
+     * Parse the labels to label map
+     *
+     * @param labels The labels format by 'key1=value1&key2=value2...'
+     * @return The label map of labels
+     */
+    public static Map<String, String> parseLabels(String labels) {
+        return parseLabels(labels, new LinkedHashMap<>());
+    }
+
+    /**
+     * Parse the labels to label map
+     *
+     * @param labels The labels format by 'key1=value1&key2=value2...'
+     * @return The label map of labels
+     */
+    public static Map<String, String> parseLabels(String labels, Map<String, String> labelMap) {
+        if (labelMap == null) {
+            labelMap = new LinkedHashMap<>();
+        }
+        if (labels == null || labels.length() == 0) {
+            return labelMap;
+        }
+        String[] labelArray = labels.split(DELIMITER);
+        for (String label : labelArray) {
+            int index = label.indexOf(KEY_VALUE_DELIMITER);
+            if (index < 1 || index == label.length() - 1) {
+                throw new IllegalArgumentException("The format of option 'sink.dirty.labels' must be key=value");

Review Comment:
   > LabelUtils is only used for 'dirty.lables'?
   
   The code here is repeated with inlong.metric.labels, I abstracted it, but there is a problem with the error message, I will deal with it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] yunqingmoswu commented on pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on PR #6618:
URL: https://github.com/apache/inlong/pull/6618#issuecomment-1328646298

   > @yunqingmoswu please git rebase master
   
   done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] gong commented on pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink

Posted by GitBox <gi...@apache.org>.
gong commented on PR #6618:
URL: https://github.com/apache/inlong/pull/6618#issuecomment-1328611609

   please git rebase master


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org