You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/05/27 03:58:29 UTC
[GitHub] [incubator-inlong] woofyzhao opened a new pull request, #4409: [INLONG-4408][Manager] Add iceberg and elasticsearch sink load node
woofyzhao opened a new pull request, #4409:
URL: https://github.com/apache/incubator-inlong/pull/4409
Feature #4408
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] woofyzhao commented on a diff in pull request #4409: [INLONG-4408][Manager][Sort] Add Iceberg sink load node
Posted by GitBox <gi...@apache.org>.
woofyzhao commented on code in PR #4409:
URL: https://github.com/apache/incubator-inlong/pull/4409#discussion_r884561699
##########
inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/pojo/sink/iceberg/IcebergSinkDTO.java:
##########
@@ -63,15 +60,6 @@ public class IcebergSinkDTO {
@ApiModelProperty("File format, support: Parquet, Orc, Avro")
private String fileFormat;
Review Comment:
good idea, will consider in another pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] woofyzhao commented on a diff in pull request #4409: [INLONG-4408][Manager][Sort] Add Iceberg sink load node
Posted by GitBox <gi...@apache.org>.
woofyzhao commented on code in PR #4409:
URL: https://github.com/apache/incubator-inlong/pull/4409#discussion_r884561699
##########
inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/pojo/sink/iceberg/IcebergSinkDTO.java:
##########
@@ -63,15 +60,6 @@ public class IcebergSinkDTO {
@ApiModelProperty("File format, support: Parquet, Orc, Avro")
private String fileFormat;
Review Comment:
good idea, will consider in another pr with other sink types in batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] woofyzhao commented on a diff in pull request #4409: [INLONG-4408][Manager][Sort] Add Iceberg sink load node
Posted by GitBox <gi...@apache.org>.
woofyzhao commented on code in PR #4409:
URL: https://github.com/apache/incubator-inlong/pull/4409#discussion_r885164220
##########
inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/pojo/sink/iceberg/IcebergSinkDTO.java:
##########
@@ -58,7 +59,7 @@ public class IcebergSinkDTO {
private String dataPath;
@ApiModelProperty("File format, support: Parquet, Orc, Avro")
- private String fileFormat;
+ private FileFormat fileFormat;
Review Comment:
Already reverted. No change in this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] thexiay commented on a diff in pull request #4409: [INLONG-4408][Manager] Add Iceberg and Elasticsearch sink load node
Posted by GitBox <gi...@apache.org>.
thexiay commented on code in PR #4409:
URL: https://github.com/apache/incubator-inlong/pull/4409#discussion_r884391664
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/HbaseLoadNode.java:
##########
@@ -82,7 +82,7 @@ public HbaseLoadNode(@JsonProperty("id") String id,
@JsonProperty("sinkParallelism") Integer sinkParallelism,
@JsonProperty("properties") Map<String, String> properties,
@JsonProperty("tableName") String tableName,
- @JsonProperty("nameSpace") String nameSpace,
+ @JsonProperty("namespace") String namespace,
Review Comment:
why here change hbase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] woofyzhao commented on a diff in pull request #4409: [INLONG-4408][Manager][Sort] Add Iceberg sink load node
Posted by GitBox <gi...@apache.org>.
woofyzhao commented on code in PR #4409:
URL: https://github.com/apache/incubator-inlong/pull/4409#discussion_r885164220
##########
inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/pojo/sink/iceberg/IcebergSinkDTO.java:
##########
@@ -58,7 +59,7 @@ public class IcebergSinkDTO {
private String dataPath;
@ApiModelProperty("File format, support: Parquet, Orc, Avro")
- private String fileFormat;
+ private FileFormat fileFormat;
Review Comment:
Reverted. No change in this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] gong commented on a diff in pull request #4409: [INLONG-4408][Manager] Add iceberg and elasticsearch sink load node
Posted by GitBox <gi...@apache.org>.
gong commented on code in PR #4409:
URL: https://github.com/apache/incubator-inlong/pull/4409#discussion_r883547848
##########
inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/pojo/sink/iceberg/IcebergSinkDTO.java:
##########
@@ -63,15 +60,6 @@ public class IcebergSinkDTO {
@ApiModelProperty("File format, support: Parquet, Orc, Avro")
private String fileFormat;
Review Comment:
suggest use enum `org.apache.inlong.manager.common.enums.FileFormat`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] gong commented on a diff in pull request #4409: [INLONG-4408][Manager] Add iceberg and elasticsearch sink load node
Posted by GitBox <gi...@apache.org>.
gong commented on code in PR #4409:
URL: https://github.com/apache/incubator-inlong/pull/4409#discussion_r883540343
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/constant/ElasticsearchConstant.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.protocol.constant;
+
+/**
+ * Postgres options constant
+ */
Review Comment:
describe error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] woofyzhao commented on a diff in pull request #4409: [INLONG-4408][Manager][Sort] Add Iceberg sink load node
Posted by GitBox <gi...@apache.org>.
woofyzhao commented on code in PR #4409:
URL: https://github.com/apache/incubator-inlong/pull/4409#discussion_r884474676
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/HbaseLoadNode.java:
##########
@@ -82,7 +82,7 @@ public HbaseLoadNode(@JsonProperty("id") String id,
@JsonProperty("sinkParallelism") Integer sinkParallelism,
@JsonProperty("properties") Map<String, String> properties,
@JsonProperty("tableName") String tableName,
- @JsonProperty("nameSpace") String nameSpace,
+ @JsonProperty("namespace") String namespace,
Review Comment:
to be consistent across the project
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] ciscozhou commented on a diff in pull request #4409: [INLONG-4408][Manager][Sort] Add Iceberg sink load node
Posted by GitBox <gi...@apache.org>.
ciscozhou commented on code in PR #4409:
URL: https://github.com/apache/incubator-inlong/pull/4409#discussion_r885115871
##########
inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/pojo/sink/iceberg/IcebergSinkDTO.java:
##########
@@ -58,7 +59,7 @@ public class IcebergSinkDTO {
private String dataPath;
@ApiModelProperty("File format, support: Parquet, Orc, Avro")
- private String fileFormat;
+ private FileFormat fileFormat;
Review Comment:
If the type of `fileFormat` changed to `Enum`, the `getFromJson()` method may be lost `fileFormat` field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] healchow merged pull request #4409: [INLONG-4408][Manager][Sort] Add Iceberg sink load node
Posted by GitBox <gi...@apache.org>.
healchow merged PR #4409:
URL: https://github.com/apache/incubator-inlong/pull/4409
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] gong commented on a diff in pull request #4409: [INLONG-4408][Manager] Add iceberg and elasticsearch sink load node
Posted by GitBox <gi...@apache.org>.
gong commented on code in PR #4409:
URL: https://github.com/apache/incubator-inlong/pull/4409#discussion_r883544779
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/LoadNode.java:
##########
@@ -60,7 +62,9 @@
@JsonSubTypes.Type(value = ClickHouseLoadNode.class, name = "clickHouseLoad"),
@JsonSubTypes.Type(value = SqlServerLoadNode.class, name = "sqlserverLoad"),
@JsonSubTypes.Type(value = TDSQLPostgresLoadNode.class, name = "tdsqlPostgresLoad"),
- @JsonSubTypes.Type(value = MySqlLoadNode.class, name = "mysqlLoad")
+ @JsonSubTypes.Type(value = MySqlLoadNode.class, name = "mysqlLoad"),
+ @JsonSubTypes.Type(value = IcebergLoadNode.class, name = "icebergLoad"),
+ @JsonSubTypes.Type(value = ElasticsearchLoadNode.class, name = "elasticsearchLoad")
Review Comment:
`Node` add @JsonSubTypes.Type(value = ElasticsearchLoadNode.class, name = "elasticsearchLoad"). And a UT in inlong-sort/sort-common/src/test/java/org/apache/inlong/sort/protocol/node/load.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org