You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "vinlee19 (via GitHub)" <gi...@apache.org> on 2024/07/15 11:48:44 UTC
[PR] [wip]support synchronizing the Geometry data type in MySQL. [doris-flink-connector]
vinlee19 opened a new pull request, #431:
URL: https://github.com/apache/doris-flink-connector/pull/431
# Proposed changes
Issue Number: close #xxx
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [wip]support synchronizing the Geometry data type in MySQL. [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #431:
URL: https://github.com/apache/doris-flink-connector/pull/431#discussion_r1682398544
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mysql/MysqlType.java:
##########
@@ -92,10 +92,22 @@ public class MysqlType {
private static final String JSON = "JSON";
private static final String ENUM = "ENUM";
private static final String SET = "SET";
+ private static final String GEOMETRY = "GEOMETRY";
+ private static final String POINT = "POINT";
+ private static final String LINESTRING = "LINESTRING";
+ private static final String POLYGON = "POLYGON";
+ private static final String MULTIPOINT = "MULTIPOINT";
+ private static final String MULTILINESTRING = "MULTILINESTRING";
+ private static final String MULTIPOLYGON = "MULTIPOLYGON";
+ private static final String GEOMETRYCOLLECTION = "GEOMETRYCOLLECTION";
public static String toDorisType(String type, Integer length, Integer scale) {
switch (type.toUpperCase()) {
case BIT:
+ if (length > 1) {
+ return DorisType.STRING;
+ }
+ return DorisType.BOOLEAN;
Review Comment:
What is the purpose of adding this judgment here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [wip]support synchronizing the Geometry data type in MySQL. [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #431:
URL: https://github.com/apache/doris-flink-connector/pull/431#discussion_r1682401522
##########
flink-doris-connector/pom.xml:
##########
@@ -204,6 +204,19 @@ under the License.
</exclusions>
</dependency>
+ <!-- geometry dependencies -->
+ <dependency>
+ <groupId>com.esri.geometry</groupId>
+ <artifactId>esri-geometry-api</artifactId>
+ <version>${geometry.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-core</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
Review Comment:
I see that cdc has this dependency. Can we not add this pom?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [wip]support synchronizing the Geometry data type in MySQL. [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #431:
URL: https://github.com/apache/doris-flink-connector/pull/431#discussion_r1682402205
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/deserialize/DorisJsonDebeziumDeserializationSchema.java:
##########
@@ -190,6 +201,28 @@ private JsonNode convertToJson(Schema schema, Object value) throws DorisExceptio
}
}
+ protected JsonNode convertMySQLGeometryToJson(Struct geometryStruct) {
+ try {
+ byte[] wkb = geometryStruct.getBytes("wkb");
+ String geoJson = OGCGeometry.fromBinary(ByteBuffer.wrap(wkb)).asGeoJson();
+ JsonNode originGeoNode = objectMapper.readTree(geoJson);
+ Optional<Integer> srid = Optional.ofNullable(geometryStruct.getInt32("srid"));
+ Map<String, Object> geometryInfo = new HashMap<>();
+ String geometryType = originGeoNode.get("type").asText();
+ geometryInfo.put("type", geometryType);
+ if (geometryType.equals("GeometryCollection")) {
+ geometryInfo.put("geometries", originGeoNode.get("geometries"));
+ } else {
+ geometryInfo.put("coordinates", originGeoNode.get("coordinates"));
+ }
+ geometryInfo.put("srid", srid.orElse(0));
+ return objectMapper.valueToTree(geometryInfo);
+ } catch (Exception e) {
+ throw new IllegalArgumentException(
+ String.format("Failed to convert %s to geometry JSON.", geometryStruct), e);
+ }
+ }
Review Comment:
It is recommended to add a UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [wip]support synchronizing the Geometry data type in MySQL. [doris-flink-connector]
Posted by "vinlee19 (via GitHub)" <gi...@apache.org>.
vinlee19 closed pull request #431: [wip]support synchronizing the Geometry data type in MySQL.
URL: https://github.com/apache/doris-flink-connector/pull/431
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [wip]support synchronizing the Geometry data type in MySQL. [doris-flink-connector]
Posted by "vinlee19 (via GitHub)" <gi...@apache.org>.
vinlee19 commented on code in PR #431:
URL: https://github.com/apache/doris-flink-connector/pull/431#discussion_r1682413794
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mysql/MysqlType.java:
##########
@@ -92,10 +92,22 @@ public class MysqlType {
private static final String JSON = "JSON";
private static final String ENUM = "ENUM";
private static final String SET = "SET";
+ private static final String GEOMETRY = "GEOMETRY";
+ private static final String POINT = "POINT";
+ private static final String LINESTRING = "LINESTRING";
+ private static final String POLYGON = "POLYGON";
+ private static final String MULTIPOINT = "MULTIPOINT";
+ private static final String MULTILINESTRING = "MULTILINESTRING";
+ private static final String MULTIPOLYGON = "MULTIPOLYGON";
+ private static final String GEOMETRYCOLLECTION = "GEOMETRYCOLLECTION";
public static String toDorisType(String type, Integer length, Integer scale) {
switch (type.toUpperCase()) {
case BIT:
+ if (length > 1) {
+ return DorisType.STRING;
+ }
+ return DorisType.BOOLEAN;
Review Comment:
when the length of bit is greater than 1,it will convert to a string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org