You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "DongLiang-0 (via GitHub)" <gi...@apache.org> on 2023/12/14 09:17:48 UTC
[PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
DongLiang-0 opened a new pull request, #272:
URL: https://github.com/apache/doris-flink-connector/pull/272
# Proposed changes
Formatting step two, format sink.
Related pr: https://github.com/apache/doris-flink-connector/pull/270
## Problem Summary:
In order to format the code more conveniently, we have introduced the `spotless-maven-plugin` plug-in. You can perform code style checking by executing `mvn spotless:check`, and format the code by executing `mvn spotless:apply`.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "DongLiang-0 (via GitHub)" <gi...@apache.org>.
DongLiang-0 commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1426450072
##########
flink-doris-connector/pom.xml:
##########
@@ -217,12 +217,6 @@ under the License.
<version>2.13.3</version>
</dependency>
- <dependency>
- <groupId>com.google.guava</groupId>
- <artifactId>guava</artifactId>
- <version>31.1-jre</version>
- </dependency>
Review Comment:
In flink's coding style, need to use `flink-shaded-guava` instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1426791621
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mysql/MysqlDatabaseSync.java:
##########
@@ -186,8 +189,8 @@ public DataStreamSource<String> buildCdcSource(StreamExecutionEnvironment env) {
Properties jdbcProperties = new Properties();
Properties debeziumProperties = new Properties();
- //date to string
- debeziumProperties.putAll(DateToStringConverter.DEFAULT_PROPS);
+ // date to string
+ debeziumProperties.putAll(DateToStringConverter.defaultProps);
Review Comment:
same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1427509096
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/table/DorisRowDataJdbcLookupFunction.java:
##########
@@ -17,20 +17,22 @@
package org.apache.doris.flink.table;
-import org.apache.doris.flink.cfg.DorisLookupOptions;
-import org.apache.doris.flink.cfg.DorisOptions;
-import org.apache.doris.flink.lookup.DorisJdbcLookupReader;
-import org.apache.doris.flink.lookup.DorisLookupReader;
-import org.apache.doris.flink.lookup.LookupSchema;
import org.apache.flink.annotation.VisibleForTesting;
-import com.google.common.cache.Cache;
-import com.google.common.cache.CacheBuilder;
import org.apache.flink.table.data.GenericRowData;
import org.apache.flink.table.data.RowData;
import org.apache.flink.table.functions.FunctionContext;
import org.apache.flink.table.functions.TableFunction;
import org.apache.flink.table.types.DataType;
import org.apache.flink.util.Preconditions;
+
+import org.apache.flink.shaded.guava30.com.google.common.cache.Cache;
+import org.apache.flink.shaded.guava30.com.google.common.cache.CacheBuilder;
Review Comment:
Need to modify here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1427509499
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/table/DorisRowDataLookupFunction.java:
##########
@@ -17,6 +17,16 @@
package org.apache.doris.flink.table;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.functions.FunctionContext;
+import org.apache.flink.table.functions.TableFunction;
+import org.apache.flink.table.types.DataType;
+
+import org.apache.flink.shaded.guava30.com.google.common.cache.Cache;
+import org.apache.flink.shaded.guava30.com.google.common.cache.CacheBuilder;
Review Comment:
the same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "DongLiang-0 (via GitHub)" <gi...@apache.org>.
DongLiang-0 commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1426450072
##########
flink-doris-connector/pom.xml:
##########
@@ -217,12 +217,6 @@ under the License.
<version>2.13.3</version>
</dependency>
- <dependency>
- <groupId>com.google.guava</groupId>
- <artifactId>guava</artifactId>
- <version>31.1-jre</version>
- </dependency>
Review Comment:
In flink's coding style, need to use `flink-shaded-guava` replace.
What is used here is `org.apache.flink.shaded.guava30` in `flink-sql-connector-mysql-cdc` version 2.4.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "DongLiang-0 (via GitHub)" <gi...@apache.org>.
DongLiang-0 commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1427498082
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/postgres/PostgresDateConverter.java:
##########
@@ -39,19 +39,23 @@ public class PostgresDateConverter implements CustomConverter<SchemaBuilder, Rel
private DateTimeFormatter dateFormatter = DateTimeFormatter.ISO_DATE;
private DateTimeFormatter timestampFormatter = DateTimeFormatter.ISO_DATE_TIME;
- public static Properties DEFAULT_PROPS = new Properties();
+ public static Properties defaultProps = new Properties();
Review Comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1426795501
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/sqlserver/SqlServerDateConverter.java:
##########
@@ -34,19 +35,23 @@ public class SqlServerDateConverter implements CustomConverter<SchemaBuilder, Re
private DateTimeFormatter dateFormatter = DateTimeFormatter.ISO_DATE;
private DateTimeFormatter timestampFormatter = DateTimeFormatter.ISO_DATE_TIME;
- public static Properties DEFAULT_PROPS = new Properties();
+ public static Properties defaultProps = new Properties();
Review Comment:
same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba merged PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1426792876
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/oracle/OracleDatabaseSync.java:
##########
@@ -128,11 +140,13 @@ public DataStreamSource<String> buildCdcSource(StreamExecutionEnvironment env) {
Preconditions.checkNotNull(databaseName, "database-name in oracle is required");
Preconditions.checkNotNull(schemaName, "schema-name in oracle is required");
String tableName = config.get(OracleSourceOptions.TABLE_NAME);
- //When debezium incrementally reads, it will be judged based on regexp_like.
- //When the regular length exceeds 512, an error will be reported, like ORA-12733: regular expression too long
- if(tableName.length() > 384){
- //max database name length 128
- tableName = StringUtils.isNullOrWhitespaceOnly(includingTables) ? ".*" : includingTables;
+ // When debezium incrementally reads, it will be judged based on regexp_like.
+ // When the regular length exceeds 512, an error will be reported, like ORA-12733: regular
+ // expression too long
Review Comment:
line break in advance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1426790141
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mysql/DateToStringConverter.java:
##########
@@ -43,23 +43,28 @@ public class DateToStringConverter implements CustomConverter<SchemaBuilder, Rel
private DateTimeFormatter timestampFormatter = DateTimeFormatter.ISO_DATE_TIME;
private ZoneId timestampZoneId = ZoneId.systemDefault();
- public static Properties DEFAULT_PROPS = new Properties();
+ public static Properties defaultProps = new Properties();
Review Comment:
Modify to static final
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1426790680
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mysql/DateToStringConverter.java:
##########
@@ -134,14 +140,16 @@ private String convertDateTime(Object input) {
private String convertTimestamp(Object input) {
if (input instanceof ZonedDateTime) {
- // mysql timestamp will be converted to UTC storage,and the zonedDatetime here is UTC time
+ // mysql timestamp will be converted to UTC storage,and the zonedDatetime here is UTC
+ // time
Review Comment:
newline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1426765730
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/deserialization/DorisJsonDebeziumDeserializationSchema.java:
##########
@@ -63,7 +64,9 @@ public void deserialize(SourceRecord sourceRecord, Collector<String> collector)
private JsonNode convertToJson(Schema schema, Object value) throws DorisException {
if (value == null) {
- if (schema == null) // Any schema is valid and we don't have a default, so treat this as an optional schema
+ if (schema
+ == null) // Any schema is valid and we don't have a default, so treat this as an
+ // optional schema
Review Comment:
It feels a bit strange, or put the comment above the if?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1426793375
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/oracle/OracleDateConverter.java:
##########
@@ -36,66 +37,76 @@
public class OracleDateConverter implements CustomConverter<SchemaBuilder, RelationalColumn> {
private static final Logger log = LoggerFactory.getLogger(OracleDateConverter.class);
- private static final Pattern TO_DATE = Pattern.compile("TO_DATE\\('(.*)',[ ]*'(.*)'\\)", Pattern.CASE_INSENSITIVE);
- private static final Pattern TO_TIMESTAMP = Pattern.compile("TO_TIMESTAMP\\('(.*)'\\)", Pattern.CASE_INSENSITIVE);
- private static final Pattern TIMESTAMP_OR_DATE_REGEX = Pattern.compile("^TIMESTAMP[(]\\d[)]$|^DATE$", Pattern.CASE_INSENSITIVE);
+ private static final Pattern TO_DATE =
+ Pattern.compile("TO_DATE\\('(.*)',[ ]*'(.*)'\\)", Pattern.CASE_INSENSITIVE);
+ private static final Pattern TO_TIMESTAMP =
+ Pattern.compile("TO_TIMESTAMP\\('(.*)'\\)", Pattern.CASE_INSENSITIVE);
+ private static final Pattern TIMESTAMP_OR_DATE_REGEX =
+ Pattern.compile("^TIMESTAMP[(]\\d[)]$|^DATE$", Pattern.CASE_INSENSITIVE);
private ZoneId timestampZoneId = ZoneId.systemDefault();
- public static Properties DEFAULT_PROPS = new Properties();
- private final String DATETIME_PATTERN = "yyyy-MM-dd HH:mm:ss";
- private final String DATETIMEV2_PATTERN = "yyyy-MM-dd HH:mm:ss.SSSSSS";
- private final DateTimeFormatter dateTimeV2Formatter = DateTimeFormatter.ofPattern(DATETIMEV2_PATTERN);
+ public static Properties defaultProps = new Properties();
+ private final String datetimePattern = "yyyy-MM-dd HH:mm:ss";
Review Comment:
static final
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "DongLiang-0 (via GitHub)" <gi...@apache.org>.
DongLiang-0 commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1427492860
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mysql/DateToStringConverter.java:
##########
@@ -43,23 +43,28 @@ public class DateToStringConverter implements CustomConverter<SchemaBuilder, Rel
private DateTimeFormatter timestampFormatter = DateTimeFormatter.ISO_DATE_TIME;
private ZoneId timestampZoneId = ZoneId.systemDefault();
- public static Properties DEFAULT_PROPS = new Properties();
+ public static Properties defaultProps = new Properties();
Review Comment:
That's a good idea.
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/DorisStreamLoad.java:
##########
@@ -143,7 +157,8 @@ public void abortPreCommit(String labelSuffix, long chkID) throws Exception {
LOG.info("abort for labelSuffix {}. start chkId {}.", labelSuffix, chkID);
while (true) {
try {
- // TODO: According to label abort txn. Currently, it can only be aborted based on txnid,
+ // TODO: According to label abort txn. Currently, it can only be aborted based on
+ // txnid,
Review Comment:
sure
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/deserialization/DorisJsonDebeziumDeserializationSchema.java:
##########
@@ -63,7 +64,9 @@ public void deserialize(SourceRecord sourceRecord, Collector<String> collector)
private JsonNode convertToJson(Schema schema, Object value) throws DorisException {
if (value == null) {
- if (schema == null) // Any schema is valid and we don't have a default, so treat this as an optional schema
+ if (schema
+ == null) // Any schema is valid and we don't have a default, so treat this as an
+ // optional schema
Review Comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1426797806
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/DorisStreamLoad.java:
##########
@@ -143,7 +157,8 @@ public void abortPreCommit(String labelSuffix, long chkID) throws Exception {
LOG.info("abort for labelSuffix {}. start chkId {}.", labelSuffix, chkID);
while (true) {
try {
- // TODO: According to label abort txn. Currently, it can only be aborted based on txnid,
+ // TODO: According to label abort txn. Currently, it can only be aborted based on
+ // txnid,
Review Comment:
line break in advance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "DongLiang-0 (via GitHub)" <gi...@apache.org>.
DongLiang-0 commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1427493504
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/oracle/OracleDateConverter.java:
##########
@@ -36,66 +37,76 @@
public class OracleDateConverter implements CustomConverter<SchemaBuilder, RelationalColumn> {
private static final Logger log = LoggerFactory.getLogger(OracleDateConverter.class);
- private static final Pattern TO_DATE = Pattern.compile("TO_DATE\\('(.*)',[ ]*'(.*)'\\)", Pattern.CASE_INSENSITIVE);
- private static final Pattern TO_TIMESTAMP = Pattern.compile("TO_TIMESTAMP\\('(.*)'\\)", Pattern.CASE_INSENSITIVE);
- private static final Pattern TIMESTAMP_OR_DATE_REGEX = Pattern.compile("^TIMESTAMP[(]\\d[)]$|^DATE$", Pattern.CASE_INSENSITIVE);
+ private static final Pattern TO_DATE =
+ Pattern.compile("TO_DATE\\('(.*)',[ ]*'(.*)'\\)", Pattern.CASE_INSENSITIVE);
+ private static final Pattern TO_TIMESTAMP =
+ Pattern.compile("TO_TIMESTAMP\\('(.*)'\\)", Pattern.CASE_INSENSITIVE);
+ private static final Pattern TIMESTAMP_OR_DATE_REGEX =
+ Pattern.compile("^TIMESTAMP[(]\\d[)]$|^DATE$", Pattern.CASE_INSENSITIVE);
private ZoneId timestampZoneId = ZoneId.systemDefault();
- public static Properties DEFAULT_PROPS = new Properties();
- private final String DATETIME_PATTERN = "yyyy-MM-dd HH:mm:ss";
- private final String DATETIMEV2_PATTERN = "yyyy-MM-dd HH:mm:ss.SSSSSS";
- private final DateTimeFormatter dateTimeV2Formatter = DateTimeFormatter.ofPattern(DATETIMEV2_PATTERN);
+ public static Properties defaultProps = new Properties();
+ private final String datetimePattern = "yyyy-MM-dd HH:mm:ss";
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "DongLiang-0 (via GitHub)" <gi...@apache.org>.
DongLiang-0 commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1427488506
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/deserialization/DorisJsonDebeziumDeserializationSchema.java:
##########
@@ -111,65 +116,74 @@ private JsonNode convertToJson(Schema schema, Object value) throws DorisExceptio
} else if (value instanceof BigDecimal) {
return JSON_NODE_FACTORY.numberNode((BigDecimal) value);
} else {
- throw new DorisException("Invalid type for bytes type: " + value.getClass());
+ throw new DorisException(
+ "Invalid type for bytes type: " + value.getClass());
}
- case ARRAY: {
- Collection<?> collection = (Collection<?>) value;
- ArrayNode list = JSON_NODE_FACTORY.arrayNode();
- for (Object elem : collection) {
- Schema valueSchema = schema == null ? null : schema.valueSchema();
- JsonNode fieldValue = convertToJson(valueSchema, elem);
- list.add(fieldValue);
+ case ARRAY:
+ {
+ Collection<?> collection = (Collection<?>) value;
+ ArrayNode list = JSON_NODE_FACTORY.arrayNode();
+ for (Object elem : collection) {
+ Schema valueSchema = schema == null ? null : schema.valueSchema();
+ JsonNode fieldValue = convertToJson(valueSchema, elem);
+ list.add(fieldValue);
+ }
+ return list;
}
- return list;
- }
- case MAP: {
- Map<?, ?> map = (Map<?, ?>) value;
- // If true, using string keys and JSON object; if false, using non-string keys and Array-encoding
- boolean objectMode;
- if (schema == null) {
- objectMode = true;
- for (Map.Entry<?, ?> entry : map.entrySet()) {
- if (!(entry.getKey() instanceof String)) {
- objectMode = false;
- break;
+ case MAP:
+ {
+ Map<?, ?> map = (Map<?, ?>) value;
+ // If true, using string keys and JSON object; if false, using non-string
+ // keys and Array-encoding
+ boolean objectMode;
+ if (schema == null) {
+ objectMode = true;
+ for (Map.Entry<?, ?> entry : map.entrySet()) {
+ if (!(entry.getKey() instanceof String)) {
+ objectMode = false;
+ break;
+ }
}
+ } else {
+ objectMode = schema.keySchema().type() == Schema.Type.STRING;
}
- } else {
- objectMode = schema.keySchema().type() == Schema.Type.STRING;
- }
- ObjectNode obj = null;
- ArrayNode list = null;
- if (objectMode) {
- obj = JSON_NODE_FACTORY.objectNode();
- } else {
- list = JSON_NODE_FACTORY.arrayNode();
- }
- for (Map.Entry<?, ?> entry : map.entrySet()) {
- Schema keySchema = schema == null ? null : schema.keySchema();
- Schema valueSchema = schema == null ? null : schema.valueSchema();
- JsonNode mapKey = convertToJson(keySchema, entry.getKey());
- JsonNode mapValue = convertToJson(valueSchema, entry.getValue());
-
+ ObjectNode obj = null;
+ ArrayNode list = null;
if (objectMode) {
- obj.set(mapKey.asText(), mapValue);
+ obj = JSON_NODE_FACTORY.objectNode();
} else {
- list.add(JSON_NODE_FACTORY.arrayNode().add(mapKey).add(mapValue));
+ list = JSON_NODE_FACTORY.arrayNode();
}
+ for (Map.Entry<?, ?> entry : map.entrySet()) {
+ Schema keySchema = schema == null ? null : schema.keySchema();
+ Schema valueSchema = schema == null ? null : schema.valueSchema();
+ JsonNode mapKey = convertToJson(keySchema, entry.getKey());
+ JsonNode mapValue = convertToJson(valueSchema, entry.getValue());
+
+ if (objectMode) {
+ obj.set(mapKey.asText(), mapValue);
Review Comment:
Notice
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/deserialization/DorisJsonDebeziumDeserializationSchema.java:
##########
@@ -111,65 +116,74 @@ private JsonNode convertToJson(Schema schema, Object value) throws DorisExceptio
} else if (value instanceof BigDecimal) {
return JSON_NODE_FACTORY.numberNode((BigDecimal) value);
} else {
- throw new DorisException("Invalid type for bytes type: " + value.getClass());
+ throw new DorisException(
+ "Invalid type for bytes type: " + value.getClass());
}
- case ARRAY: {
- Collection<?> collection = (Collection<?>) value;
- ArrayNode list = JSON_NODE_FACTORY.arrayNode();
- for (Object elem : collection) {
- Schema valueSchema = schema == null ? null : schema.valueSchema();
- JsonNode fieldValue = convertToJson(valueSchema, elem);
- list.add(fieldValue);
+ case ARRAY:
+ {
+ Collection<?> collection = (Collection<?>) value;
+ ArrayNode list = JSON_NODE_FACTORY.arrayNode();
+ for (Object elem : collection) {
+ Schema valueSchema = schema == null ? null : schema.valueSchema();
+ JsonNode fieldValue = convertToJson(valueSchema, elem);
+ list.add(fieldValue);
+ }
+ return list;
}
- return list;
- }
- case MAP: {
- Map<?, ?> map = (Map<?, ?>) value;
- // If true, using string keys and JSON object; if false, using non-string keys and Array-encoding
- boolean objectMode;
- if (schema == null) {
- objectMode = true;
- for (Map.Entry<?, ?> entry : map.entrySet()) {
- if (!(entry.getKey() instanceof String)) {
- objectMode = false;
- break;
+ case MAP:
+ {
+ Map<?, ?> map = (Map<?, ?>) value;
+ // If true, using string keys and JSON object; if false, using non-string
+ // keys and Array-encoding
+ boolean objectMode;
+ if (schema == null) {
+ objectMode = true;
+ for (Map.Entry<?, ?> entry : map.entrySet()) {
+ if (!(entry.getKey() instanceof String)) {
+ objectMode = false;
+ break;
+ }
}
+ } else {
+ objectMode = schema.keySchema().type() == Schema.Type.STRING;
}
- } else {
- objectMode = schema.keySchema().type() == Schema.Type.STRING;
- }
- ObjectNode obj = null;
- ArrayNode list = null;
- if (objectMode) {
- obj = JSON_NODE_FACTORY.objectNode();
- } else {
- list = JSON_NODE_FACTORY.arrayNode();
- }
- for (Map.Entry<?, ?> entry : map.entrySet()) {
- Schema keySchema = schema == null ? null : schema.keySchema();
- Schema valueSchema = schema == null ? null : schema.valueSchema();
- JsonNode mapKey = convertToJson(keySchema, entry.getKey());
- JsonNode mapValue = convertToJson(valueSchema, entry.getValue());
-
+ ObjectNode obj = null;
+ ArrayNode list = null;
if (objectMode) {
- obj.set(mapKey.asText(), mapValue);
+ obj = JSON_NODE_FACTORY.objectNode();
Review Comment:
As expected, due to the previous code formatting, the code corresponding to line `155` is on line `164`.
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mysql/DateToStringConverter.java:
##########
@@ -43,23 +43,28 @@ public class DateToStringConverter implements CustomConverter<SchemaBuilder, Rel
private DateTimeFormatter timestampFormatter = DateTimeFormatter.ISO_DATE_TIME;
private ZoneId timestampZoneId = ZoneId.systemDefault();
- public static Properties DEFAULT_PROPS = new Properties();
+ public static Properties defaultProps = new Properties();
Review Comment:
That's a good idea.
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/oracle/OracleDatabaseSync.java:
##########
@@ -128,11 +140,13 @@ public DataStreamSource<String> buildCdcSource(StreamExecutionEnvironment env) {
Preconditions.checkNotNull(databaseName, "database-name in oracle is required");
Preconditions.checkNotNull(schemaName, "schema-name in oracle is required");
String tableName = config.get(OracleSourceOptions.TABLE_NAME);
- //When debezium incrementally reads, it will be judged based on regexp_like.
- //When the regular length exceeds 512, an error will be reported, like ORA-12733: regular expression too long
- if(tableName.length() > 384){
- //max database name length 128
- tableName = StringUtils.isNullOrWhitespaceOnly(includingTables) ? ".*" : includingTables;
+ // When debezium incrementally reads, it will be judged based on regexp_like.
+ // When the regular length exceeds 512, an error will be reported, like ORA-12733: regular
+ // expression too long
Review Comment:
sure
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/DorisStreamLoad.java:
##########
@@ -143,7 +157,8 @@ public void abortPreCommit(String labelSuffix, long chkID) throws Exception {
LOG.info("abort for labelSuffix {}. start chkId {}.", labelSuffix, chkID);
while (true) {
try {
- // TODO: According to label abort txn. Currently, it can only be aborted based on txnid,
+ // TODO: According to label abort txn. Currently, it can only be aborted based on
+ // txnid,
Review Comment:
sure
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/deserialization/DorisJsonDebeziumDeserializationSchema.java:
##########
@@ -63,7 +64,9 @@ public void deserialize(SourceRecord sourceRecord, Collector<String> collector)
private JsonNode convertToJson(Schema schema, Object value) throws DorisException {
if (value == null) {
- if (schema == null) // Any schema is valid and we don't have a default, so treat this as an optional schema
+ if (schema
+ == null) // Any schema is valid and we don't have a default, so treat this as an
+ // optional schema
Review Comment:
sure
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/oracle/OracleDateConverter.java:
##########
@@ -36,66 +37,76 @@
public class OracleDateConverter implements CustomConverter<SchemaBuilder, RelationalColumn> {
private static final Logger log = LoggerFactory.getLogger(OracleDateConverter.class);
- private static final Pattern TO_DATE = Pattern.compile("TO_DATE\\('(.*)',[ ]*'(.*)'\\)", Pattern.CASE_INSENSITIVE);
- private static final Pattern TO_TIMESTAMP = Pattern.compile("TO_TIMESTAMP\\('(.*)'\\)", Pattern.CASE_INSENSITIVE);
- private static final Pattern TIMESTAMP_OR_DATE_REGEX = Pattern.compile("^TIMESTAMP[(]\\d[)]$|^DATE$", Pattern.CASE_INSENSITIVE);
+ private static final Pattern TO_DATE =
+ Pattern.compile("TO_DATE\\('(.*)',[ ]*'(.*)'\\)", Pattern.CASE_INSENSITIVE);
+ private static final Pattern TO_TIMESTAMP =
+ Pattern.compile("TO_TIMESTAMP\\('(.*)'\\)", Pattern.CASE_INSENSITIVE);
+ private static final Pattern TIMESTAMP_OR_DATE_REGEX =
+ Pattern.compile("^TIMESTAMP[(]\\d[)]$|^DATE$", Pattern.CASE_INSENSITIVE);
private ZoneId timestampZoneId = ZoneId.systemDefault();
- public static Properties DEFAULT_PROPS = new Properties();
- private final String DATETIME_PATTERN = "yyyy-MM-dd HH:mm:ss";
- private final String DATETIMEV2_PATTERN = "yyyy-MM-dd HH:mm:ss.SSSSSS";
- private final DateTimeFormatter dateTimeV2Formatter = DateTimeFormatter.ofPattern(DATETIMEV2_PATTERN);
+ public static Properties defaultProps = new Properties();
+ private final String datetimePattern = "yyyy-MM-dd HH:mm:ss";
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1427509411
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/table/DorisRowDataLookupFunction.java:
##########
@@ -17,6 +17,16 @@
package org.apache.doris.flink.table;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.functions.FunctionContext;
+import org.apache.flink.table.functions.TableFunction;
+import org.apache.flink.table.types.DataType;
+
+import org.apache.flink.shaded.guava30.com.google.common.cache.Cache;
+import org.apache.flink.shaded.guava30.com.google.common.cache.CacheBuilder;
Review Comment:
the same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1426482031
##########
flink-doris-connector/pom.xml:
##########
@@ -217,12 +217,6 @@ under the License.
<version>2.13.3</version>
</dependency>
- <dependency>
- <groupId>com.google.guava</groupId>
- <artifactId>guava</artifactId>
- <version>31.1-jre</version>
- </dependency>
Review Comment:
It is recommended to remove the verification of flink-shaded-guava first, otherwise there may be problems in other Flink versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1426793711
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/oracle/OracleDateConverter.java:
##########
@@ -105,28 +116,32 @@ private String convertStringTimestamp(String data) {
final Matcher toTimestampMatcher = TO_TIMESTAMP.matcher(data);
if (toTimestampMatcher.matches()) {
String dateText = toTimestampMatcher.group(1);
- dateTime = LocalDateTime.from(TIMESTAMP_FORMATTER.parse(completeMilliseconds(dateText.trim())));
+ dateTime =
+ LocalDateTime.from(
+ TIMESTAMP_FORMATTER.parse(completeMilliseconds(dateText.trim())));
return dateTimeV2Formatter.format(dateTime.atZone(timestampZoneId));
}
final Matcher toDateMatcher = TO_DATE.matcher(data);
if (toDateMatcher.matches()) {
String date = toDateMatcher.group(1);
- dateTime = LocalDateTime.from(TIMESTAMP_FORMATTER.parse(completeMilliseconds(date.trim())));
+ dateTime =
+ LocalDateTime.from(
+ TIMESTAMP_FORMATTER.parse(completeMilliseconds(date.trim())));
return dateTimeV2Formatter.format(dateTime.atZone(timestampZoneId));
}
return null;
}
private String completeMilliseconds(String stringValue) {
- if(stringValue.length() == DATETIMEV2_PATTERN.length()){
+ if (stringValue.length() == datetimeV2Pattern.length()) {
Review Comment:
same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "DongLiang-0 (via GitHub)" <gi...@apache.org>.
DongLiang-0 commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1427488555
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/deserialization/DorisJsonDebeziumDeserializationSchema.java:
##########
@@ -111,65 +116,74 @@ private JsonNode convertToJson(Schema schema, Object value) throws DorisExceptio
} else if (value instanceof BigDecimal) {
return JSON_NODE_FACTORY.numberNode((BigDecimal) value);
} else {
- throw new DorisException("Invalid type for bytes type: " + value.getClass());
+ throw new DorisException(
+ "Invalid type for bytes type: " + value.getClass());
}
- case ARRAY: {
- Collection<?> collection = (Collection<?>) value;
- ArrayNode list = JSON_NODE_FACTORY.arrayNode();
- for (Object elem : collection) {
- Schema valueSchema = schema == null ? null : schema.valueSchema();
- JsonNode fieldValue = convertToJson(valueSchema, elem);
- list.add(fieldValue);
+ case ARRAY:
+ {
+ Collection<?> collection = (Collection<?>) value;
+ ArrayNode list = JSON_NODE_FACTORY.arrayNode();
+ for (Object elem : collection) {
+ Schema valueSchema = schema == null ? null : schema.valueSchema();
+ JsonNode fieldValue = convertToJson(valueSchema, elem);
+ list.add(fieldValue);
+ }
+ return list;
}
- return list;
- }
- case MAP: {
- Map<?, ?> map = (Map<?, ?>) value;
- // If true, using string keys and JSON object; if false, using non-string keys and Array-encoding
- boolean objectMode;
- if (schema == null) {
- objectMode = true;
- for (Map.Entry<?, ?> entry : map.entrySet()) {
- if (!(entry.getKey() instanceof String)) {
- objectMode = false;
- break;
+ case MAP:
+ {
+ Map<?, ?> map = (Map<?, ?>) value;
+ // If true, using string keys and JSON object; if false, using non-string
+ // keys and Array-encoding
+ boolean objectMode;
+ if (schema == null) {
+ objectMode = true;
+ for (Map.Entry<?, ?> entry : map.entrySet()) {
+ if (!(entry.getKey() instanceof String)) {
+ objectMode = false;
+ break;
+ }
}
+ } else {
+ objectMode = schema.keySchema().type() == Schema.Type.STRING;
}
- } else {
- objectMode = schema.keySchema().type() == Schema.Type.STRING;
- }
- ObjectNode obj = null;
- ArrayNode list = null;
- if (objectMode) {
- obj = JSON_NODE_FACTORY.objectNode();
- } else {
- list = JSON_NODE_FACTORY.arrayNode();
- }
- for (Map.Entry<?, ?> entry : map.entrySet()) {
- Schema keySchema = schema == null ? null : schema.keySchema();
- Schema valueSchema = schema == null ? null : schema.valueSchema();
- JsonNode mapKey = convertToJson(keySchema, entry.getKey());
- JsonNode mapValue = convertToJson(valueSchema, entry.getValue());
-
+ ObjectNode obj = null;
+ ArrayNode list = null;
if (objectMode) {
- obj.set(mapKey.asText(), mapValue);
+ obj = JSON_NODE_FACTORY.objectNode();
Review Comment:
As expected, due to the previous code formatting, the code corresponding to line `155` is on line `164`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "DongLiang-0 (via GitHub)" <gi...@apache.org>.
DongLiang-0 commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1427497219
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/deserialization/DorisJsonDebeziumDeserializationSchema.java:
##########
@@ -111,65 +116,74 @@ private JsonNode convertToJson(Schema schema, Object value) throws DorisExceptio
} else if (value instanceof BigDecimal) {
return JSON_NODE_FACTORY.numberNode((BigDecimal) value);
} else {
- throw new DorisException("Invalid type for bytes type: " + value.getClass());
+ throw new DorisException(
+ "Invalid type for bytes type: " + value.getClass());
}
- case ARRAY: {
- Collection<?> collection = (Collection<?>) value;
- ArrayNode list = JSON_NODE_FACTORY.arrayNode();
- for (Object elem : collection) {
- Schema valueSchema = schema == null ? null : schema.valueSchema();
- JsonNode fieldValue = convertToJson(valueSchema, elem);
- list.add(fieldValue);
+ case ARRAY:
+ {
+ Collection<?> collection = (Collection<?>) value;
+ ArrayNode list = JSON_NODE_FACTORY.arrayNode();
+ for (Object elem : collection) {
+ Schema valueSchema = schema == null ? null : schema.valueSchema();
+ JsonNode fieldValue = convertToJson(valueSchema, elem);
+ list.add(fieldValue);
+ }
+ return list;
}
- return list;
- }
- case MAP: {
- Map<?, ?> map = (Map<?, ?>) value;
- // If true, using string keys and JSON object; if false, using non-string keys and Array-encoding
- boolean objectMode;
- if (schema == null) {
- objectMode = true;
- for (Map.Entry<?, ?> entry : map.entrySet()) {
- if (!(entry.getKey() instanceof String)) {
- objectMode = false;
- break;
+ case MAP:
+ {
+ Map<?, ?> map = (Map<?, ?>) value;
+ // If true, using string keys and JSON object; if false, using non-string
+ // keys and Array-encoding
+ boolean objectMode;
+ if (schema == null) {
+ objectMode = true;
+ for (Map.Entry<?, ?> entry : map.entrySet()) {
+ if (!(entry.getKey() instanceof String)) {
+ objectMode = false;
+ break;
+ }
}
+ } else {
+ objectMode = schema.keySchema().type() == Schema.Type.STRING;
}
- } else {
- objectMode = schema.keySchema().type() == Schema.Type.STRING;
- }
- ObjectNode obj = null;
- ArrayNode list = null;
- if (objectMode) {
- obj = JSON_NODE_FACTORY.objectNode();
- } else {
- list = JSON_NODE_FACTORY.arrayNode();
- }
- for (Map.Entry<?, ?> entry : map.entrySet()) {
- Schema keySchema = schema == null ? null : schema.keySchema();
- Schema valueSchema = schema == null ? null : schema.valueSchema();
- JsonNode mapKey = convertToJson(keySchema, entry.getKey());
- JsonNode mapValue = convertToJson(valueSchema, entry.getValue());
-
+ ObjectNode obj = null;
+ ArrayNode list = null;
if (objectMode) {
- obj.set(mapKey.asText(), mapValue);
+ obj = JSON_NODE_FACTORY.objectNode();
Review Comment:
Due to code formatting reasons, the actual line 153 corresponds to line 164.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "DongLiang-0 (via GitHub)" <gi...@apache.org>.
DongLiang-0 commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1427495534
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/sqlserver/SqlServerDateConverter.java:
##########
@@ -34,19 +35,23 @@ public class SqlServerDateConverter implements CustomConverter<SchemaBuilder, Re
private DateTimeFormatter dateFormatter = DateTimeFormatter.ISO_DATE;
private DateTimeFormatter timestampFormatter = DateTimeFormatter.ISO_DATE_TIME;
- public static Properties DEFAULT_PROPS = new Properties();
+ public static Properties defaultProps = new Properties();
Review Comment:
sure
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/oracle/OracleDatabaseSync.java:
##########
@@ -128,11 +140,13 @@ public DataStreamSource<String> buildCdcSource(StreamExecutionEnvironment env) {
Preconditions.checkNotNull(databaseName, "database-name in oracle is required");
Preconditions.checkNotNull(schemaName, "schema-name in oracle is required");
String tableName = config.get(OracleSourceOptions.TABLE_NAME);
- //When debezium incrementally reads, it will be judged based on regexp_like.
- //When the regular length exceeds 512, an error will be reported, like ORA-12733: regular expression too long
- if(tableName.length() > 384){
- //max database name length 128
- tableName = StringUtils.isNullOrWhitespaceOnly(includingTables) ? ".*" : includingTables;
+ // When debezium incrementally reads, it will be judged based on regexp_like.
+ // When the regular length exceeds 512, an error will be reported, like ORA-12733: regular
+ // expression too long
Review Comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "DongLiang-0 (via GitHub)" <gi...@apache.org>.
DongLiang-0 commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1427496149
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/deserialization/DorisJsonDebeziumDeserializationSchema.java:
##########
@@ -63,7 +64,9 @@ public void deserialize(SourceRecord sourceRecord, Collector<String> collector)
private JsonNode convertToJson(Schema schema, Object value) throws DorisException {
if (value == null) {
- if (schema == null) // Any schema is valid and we don't have a default, so treat this as an optional schema
+ if (schema
+ == null) // Any schema is valid and we don't have a default, so treat this as an
+ // optional schema
Review Comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1426769264
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/deserialization/DorisJsonDebeziumDeserializationSchema.java:
##########
@@ -111,65 +116,74 @@ private JsonNode convertToJson(Schema schema, Object value) throws DorisExceptio
} else if (value instanceof BigDecimal) {
return JSON_NODE_FACTORY.numberNode((BigDecimal) value);
} else {
- throw new DorisException("Invalid type for bytes type: " + value.getClass());
+ throw new DorisException(
+ "Invalid type for bytes type: " + value.getClass());
}
- case ARRAY: {
- Collection<?> collection = (Collection<?>) value;
- ArrayNode list = JSON_NODE_FACTORY.arrayNode();
- for (Object elem : collection) {
- Schema valueSchema = schema == null ? null : schema.valueSchema();
- JsonNode fieldValue = convertToJson(valueSchema, elem);
- list.add(fieldValue);
+ case ARRAY:
+ {
+ Collection<?> collection = (Collection<?>) value;
+ ArrayNode list = JSON_NODE_FACTORY.arrayNode();
+ for (Object elem : collection) {
+ Schema valueSchema = schema == null ? null : schema.valueSchema();
+ JsonNode fieldValue = convertToJson(valueSchema, elem);
+ list.add(fieldValue);
+ }
+ return list;
}
- return list;
- }
- case MAP: {
- Map<?, ?> map = (Map<?, ?>) value;
- // If true, using string keys and JSON object; if false, using non-string keys and Array-encoding
- boolean objectMode;
- if (schema == null) {
- objectMode = true;
- for (Map.Entry<?, ?> entry : map.entrySet()) {
- if (!(entry.getKey() instanceof String)) {
- objectMode = false;
- break;
+ case MAP:
+ {
+ Map<?, ?> map = (Map<?, ?>) value;
+ // If true, using string keys and JSON object; if false, using non-string
+ // keys and Array-encoding
+ boolean objectMode;
+ if (schema == null) {
+ objectMode = true;
+ for (Map.Entry<?, ?> entry : map.entrySet()) {
+ if (!(entry.getKey() instanceof String)) {
+ objectMode = false;
+ break;
+ }
}
+ } else {
+ objectMode = schema.keySchema().type() == Schema.Type.STRING;
}
- } else {
- objectMode = schema.keySchema().type() == Schema.Type.STRING;
- }
- ObjectNode obj = null;
- ArrayNode list = null;
- if (objectMode) {
- obj = JSON_NODE_FACTORY.objectNode();
- } else {
- list = JSON_NODE_FACTORY.arrayNode();
- }
- for (Map.Entry<?, ?> entry : map.entrySet()) {
- Schema keySchema = schema == null ? null : schema.keySchema();
- Schema valueSchema = schema == null ? null : schema.valueSchema();
- JsonNode mapKey = convertToJson(keySchema, entry.getKey());
- JsonNode mapValue = convertToJson(valueSchema, entry.getValue());
-
+ ObjectNode obj = null;
+ ArrayNode list = null;
if (objectMode) {
- obj.set(mapKey.asText(), mapValue);
+ obj = JSON_NODE_FACTORY.objectNode();
Review Comment:
Is this as expected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1427509668
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/table/DorisRowDataLookupFunction.java:
##########
@@ -25,14 +35,6 @@
import org.apache.doris.flink.rest.PartitionDefinition;
import org.apache.doris.flink.rest.RestService;
import org.apache.doris.flink.source.reader.DorisValueReader;
-import org.apache.flink.annotation.VisibleForTesting;
-import org.apache.flink.shaded.guava30.com.google.common.cache.Cache;
-import org.apache.flink.shaded.guava30.com.google.common.cache.CacheBuilder;
Review Comment:
the same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1426795191
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/sqlserver/SqlServerDatabaseSync.java:
##########
@@ -130,9 +141,9 @@ public DataStreamSource<String> buildCdcSource(StreamExecutionEnvironment env) {
startupOptions = StartupOptions.latest();
}
- //debezium properties set
+ // debezium properties set
Properties debeziumProperties = new Properties();
- debeziumProperties.putAll(SqlServerDateConverter.DEFAULT_PROPS);
+ debeziumProperties.putAll(SqlServerDateConverter.defaultProps);
Review Comment:
same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#issuecomment-1855961351
Thanks to @DongLiang-0 , I left some comments, PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1426794377
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/postgres/PostgresDateConverter.java:
##########
@@ -39,19 +39,23 @@ public class PostgresDateConverter implements CustomConverter<SchemaBuilder, Rel
private DateTimeFormatter dateFormatter = DateTimeFormatter.ISO_DATE;
private DateTimeFormatter timestampFormatter = DateTimeFormatter.ISO_DATE_TIME;
- public static Properties DEFAULT_PROPS = new Properties();
+ public static Properties defaultProps = new Properties();
Review Comment:
static final
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1426784633
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/serializer/RowSerializer.java:
##########
@@ -30,31 +30,34 @@
import static org.apache.doris.flink.sink.writer.LoadConstants.JSON;
/**
- * Serializer for {@link Row}.
- * Quick way to support RowSerializer on existing code
- * TODO: support original Doris to Row serializer
+ * Serializer for {@link Row}. Quick way to support RowSerializer on existing code TODO: support
+ * original Doris to Row serializer
Review Comment:
Under line wrap processing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [step2][Improve]Use flink’s checkstyle to format Sink Connector [doris-flink-connector]
Posted by "DongLiang-0 (via GitHub)" <gi...@apache.org>.
DongLiang-0 commented on code in PR #272:
URL: https://github.com/apache/doris-flink-connector/pull/272#discussion_r1426450072
##########
flink-doris-connector/pom.xml:
##########
@@ -217,12 +217,6 @@ under the License.
<version>2.13.3</version>
</dependency>
- <dependency>
- <groupId>com.google.guava</groupId>
- <artifactId>guava</artifactId>
- <version>31.1-jre</version>
- </dependency>
Review Comment:
In flink's coding style, need to use `flink-shaded-guava` replace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org