You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by "fuweng11 (via GitHub)" <gi...@apache.org> on 2023/09/20 13:12:23 UTC
[GitHub] [inlong] fuweng11 opened a new pull request, #8952: [INLONG-8951][Manager] Support for configuring built-in fields for iceberg
fuweng11 opened a new pull request, #8952:
URL: https://github.com/apache/inlong/pull/8952
### Prepare a Pull Request
- Fixes #8951
### Motivation
Support for configuring built-in fields for iceberg.
### Modifications
Support for configuring built-in fields for iceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] fuweng11 commented on a diff in pull request #8952: [INLONG-8951][Manager] Support for configuring built-in fields for iceberg
Posted by "fuweng11 (via GitHub)" <gi...@apache.org>.
fuweng11 commented on code in PR #8952:
URL: https://github.com/apache/inlong/pull/8952#discussion_r1333782646
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/StarRocksLoadNode.java:
##########
@@ -171,4 +175,20 @@ public String genTableName() {
public String getPrimaryKey() {
return primaryKey;
}
+
+ @Override
+ public List<FieldInfo> addMetaFields(List<FieldInfo> fieldInfos) {
+ List<String> fieldNames = fieldInfos.stream().map(FieldInfo::getName).collect(Collectors.toList());
+ if (!fieldNames.contains(MetaField.INLONG_DATA_TIME.name())) {
+ fieldInfos.add(0, new FieldInfo("inlong_data_time", new LongFormatInfo()));
+ }
+ return fieldInfos;
+ }
+
+ @Override
+ public List<FieldInfo> metaFields() {
+ List<FieldInfo> fieldInfos = new ArrayList<>();
+ fieldInfos.add(0, new FieldInfo("inlong_data_time", new LongFormatInfo()));
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on pull request #8952: [INLONG-8951][Manager] Support for configuring built-in fields for iceberg
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on PR #8952:
URL: https://github.com/apache/inlong/pull/8952#issuecomment-1728959223
should add this field also
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #8952: [INLONG-8951][Manager] Support for configuring built-in fields for iceberg and starrocks
Posted by "dockerzhang (via GitHub)" <gi...@apache.org>.
dockerzhang merged PR #8952:
URL: https://github.com/apache/inlong/pull/8952
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #8952: [INLONG-8951][Manager] Support for configuring built-in fields for iceberg
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on code in PR #8952:
URL: https://github.com/apache/inlong/pull/8952#discussion_r1332346931
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/StarRocksLoadNode.java:
##########
@@ -171,4 +175,20 @@ public String genTableName() {
public String getPrimaryKey() {
return primaryKey;
}
+
+ @Override
+ public List<FieldInfo> addMetaFields(List<FieldInfo> fieldInfos) {
+ List<String> fieldNames = fieldInfos.stream().map(FieldInfo::getName).collect(Collectors.toList());
+ if (!fieldNames.contains(MetaField.INLONG_DATA_TIME.name())) {
+ fieldInfos.add(0, new FieldInfo("inlong_data_time", new LongFormatInfo()));
+ }
+ return fieldInfos;
+ }
+
+ @Override
+ public List<FieldInfo> metaFields() {
+ List<FieldInfo> fieldInfos = new ArrayList<>();
+ fieldInfos.add(0, new FieldInfo("inlong_data_time", new LongFormatInfo()));
Review Comment:
INLONG_DATA_TIME can be a constant or enum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] vernedeng commented on a diff in pull request #8952: [INLONG-8951][Manager] Support for configuring built-in fields for iceberg
Posted by "vernedeng (via GitHub)" <gi...@apache.org>.
vernedeng commented on code in PR #8952:
URL: https://github.com/apache/inlong/pull/8952#discussion_r1332388168
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/StarRocksLoadNode.java:
##########
@@ -171,4 +175,20 @@ public String genTableName() {
public String getPrimaryKey() {
return primaryKey;
}
+
+ @Override
+ public List<FieldInfo> addMetaFields(List<FieldInfo> fieldInfos) {
+ List<String> fieldNames = fieldInfos.stream().map(FieldInfo::getName).collect(Collectors.toList());
+ if (!fieldNames.contains(MetaField.INLONG_DATA_TIME.name())) {
+ fieldInfos.add(0, new FieldInfo("inlong_data_time", new LongFormatInfo()));
+ }
+ return fieldInfos;
+ }
+
+ @Override
+ public List<FieldInfo> metaFields() {
+ List<FieldInfo> fieldInfos = new ArrayList<>();
+ fieldInfos.add(0, new FieldInfo("inlong_data_time", new LongFormatInfo()));
Review Comment:
There is one constant named "inlong_data_time"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] fuweng11 commented on a diff in pull request #8952: [INLONG-8951][Manager] Support for configuring built-in fields for iceberg
Posted by "fuweng11 (via GitHub)" <gi...@apache.org>.
fuweng11 commented on code in PR #8952:
URL: https://github.com/apache/inlong/pull/8952#discussion_r1332397101
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/StarRocksLoadNode.java:
##########
@@ -171,4 +175,20 @@ public String genTableName() {
public String getPrimaryKey() {
return primaryKey;
}
+
+ @Override
+ public List<FieldInfo> addMetaFields(List<FieldInfo> fieldInfos) {
+ List<String> fieldNames = fieldInfos.stream().map(FieldInfo::getName).collect(Collectors.toList());
+ if (!fieldNames.contains(MetaField.INLONG_DATA_TIME.name())) {
+ fieldInfos.add(0, new FieldInfo("inlong_data_time", new LongFormatInfo()));
+ }
+ return fieldInfos;
+ }
+
+ @Override
+ public List<FieldInfo> metaFields() {
+ List<FieldInfo> fieldInfos = new ArrayList<>();
+ fieldInfos.add(0, new FieldInfo("inlong_data_time", new LongFormatInfo()));
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on pull request #8952: [INLONG-8951][Manager] Support for configuring built-in fields for iceberg
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on PR #8952:
URL: https://github.com/apache/inlong/pull/8952#issuecomment-1728720312
please change this file also
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] vernedeng commented on a diff in pull request #8952: [INLONG-8951][Manager] Support for configuring built-in fields for iceberg
Posted by "vernedeng (via GitHub)" <gi...@apache.org>.
vernedeng commented on code in PR #8952:
URL: https://github.com/apache/inlong/pull/8952#discussion_r1333788969
##########
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sort/node/provider/IcebergProvider.java:
##########
@@ -91,4 +98,22 @@ public LoadNode createLoadNode(StreamNode nodeInfo, Map<String, StreamField> con
icebergSink.getCatalogUri(),
icebergSink.getWarehouse());
}
+
+ @Override
+ public List<StreamField> addStreamMetaFields(List<StreamField> streamFields) {
+ List<String> fieldNames = streamFields.stream().map(StreamField::getFieldName).collect(Collectors.toList());
+ if (!fieldNames.contains(MetaField.AUDIT_DATA_TIME.name())) {
+ streamFields.add(0,
+ new StreamField(0, "long", MetaField.AUDIT_DATA_TIME.name(), "date_time", null, 1,
Review Comment:
data_time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] fuweng11 commented on a diff in pull request #8952: [INLONG-8951][Manager] Support for configuring built-in fields for iceberg
Posted by "fuweng11 (via GitHub)" <gi...@apache.org>.
fuweng11 commented on code in PR #8952:
URL: https://github.com/apache/inlong/pull/8952#discussion_r1333793596
##########
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sort/node/provider/IcebergProvider.java:
##########
@@ -91,4 +98,22 @@ public LoadNode createLoadNode(StreamNode nodeInfo, Map<String, StreamField> con
icebergSink.getCatalogUri(),
icebergSink.getWarehouse());
}
+
+ @Override
+ public List<StreamField> addStreamMetaFields(List<StreamField> streamFields) {
+ List<String> fieldNames = streamFields.stream().map(StreamField::getFieldName).collect(Collectors.toList());
+ if (!fieldNames.contains(MetaField.AUDIT_DATA_TIME.name())) {
+ streamFields.add(0,
+ new StreamField(0, "long", MetaField.AUDIT_DATA_TIME.name(), "date_time", null, 1,
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org