You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/11/25 04:45:29 UTC
[GitHub] [inlong] Yizhou-Yang commented on a diff in pull request #6618: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink
Yizhou-Yang commented on code in PR #6618:
URL: https://github.com/apache/inlong/pull/6618#discussion_r1032001028
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java:
##########
@@ -177,4 +181,93 @@ public final class Constants {
.defaultValue(false)
.withDescription("Because spark do not support iceberg data type: `timestamp without time zone` and"
+ "`time`, so type conversions must be mapped to types supported by spark.");
+
+ //========================================= dirty configuration =========================================
+ public static final String DEFAULT_LABEL_IDENTIFER = "dirty-data";
+ public static final String SINK_DIRTY_PREFIX = "sink.dirty.";
+ public static final String DIRTY_PREFIX = "dirty.";
+ public static final String DIRTY_LABEL_DELIMITER = "-";
+
+ public static final ConfigOption<Boolean> DIRTY_IGNORE =
+ ConfigOptions.key("dirty.ignore")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription("Whether ignore the dirty data, default value is 'false'");
+ public static final ConfigOption<Boolean> DIRTY_SINK_ENABLE =
+ ConfigOptions.key("dirty.sink.enable")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription("Whether supports dirty data sink, default value is 'false'");
+ public static final ConfigOption<String> SINK_DIRTY_CONNECTOR =
+ ConfigOptions.key("sink.dirty.connector")
+ .stringType()
+ .noDefaultValue()
+ .withDescription("The connector of dirty sink");
+ public static final ConfigOption<String> SINK_DIRTY_FORMAT =
+ ConfigOptions.key("sink.dirty.format")
+ .stringType()
+ .defaultValue("csv")
Review Comment:
for now only supports csv and json?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org